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WELCOME 


Welcome  to  the  Health  Care  Financing  Administration's  and  Substance  Abuse  and 
Mental  Health  Services  Administration's  "Quality  Assessment  and  Improvement: 
Tools  and  Strategies  for  State  Medicaid,  Mental  Health  and  Substance  Abuse 
Agencies"  conference.  The  purpose  of  this  conference  is  to  provide  technical 
assistance  to  State  agency  management  and  staff  and  HCFA  Regional  Office  staff  on 
how  to  implement  valid  and  reliable  methods  of  overseeing  and  improving  health 
care  quality. 

Our  plenary  keynote  speaker,  Rachel  Block,  Deputy  Director  of  the  Center  for 
Medicaid  and  State  Operations,  HCFA,  will  address  the  importance  of  quality 
improvement  and  a  State  quality  strategy.  The  plenary  session  continues  with  Emma 
Hoo,  Director  of  Value  Based  Purchasing  for  Pacific  Business  Group  on  Health.  Ms. 
Hoo  will  provide  a  private  sector  perspective  on  how  to  implement  a  quality  strategy. 

Michael  Millenson  will  be  our  luncheon  speaker.  He  is  the  author  of  Demanding 
Medical  Excellence  and  a  three-time  Pulitzer  Prize  nominee  as  a  reporter  for  the 
Chicago  Tribune*  Mr.  Millenson  will  address,  from  the  private  sector  perspective,  the 
factors  that  influence  the  measurement  and  improvement  of  health  care. 

A  number  of  the  sessions  will  begin  with  an  introduction  to  the  HCFA  External 
Quality  Review  Protocols  for  the  applicable  quality  improvement  activity.  The 
session  speakers  will  provide  case  examples  of  how  they  have  implemented  the 
quality  improvement  activity.  To  facilitate  an  exchange  of  ideas,  time  has  been 
allocated  at  the  end  of  each  session  for  questions  and  answers. 

We  hope  that  you  leave  this  conference  with  information  that  will  assist  you  when 
conceptualizing  and  implementing  your  State  quality  strategy.  We  will  ask  some 
pointed  questions  in  our  conference  evaluation  to  this  effect  and  Will  follow  up  with 
an  evaluation  a  year  from  now  to  see  if  this  conference  did  assist  you  and  how  we 
can  improve  on  future  training  efforts. 

Should  you  need  any  assistance,  please  feel  free  to  contact  any  member  of  the  hotel 
or  AFYA  staff,  our  logistics  contractor. 


Have  a  great  conference! 
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AGENDA 


MONDAY,  SEPTEMBER  25,  2000 


5:00  PM  —   8:00  PM     PRE-CONFERENCE  REGISTRATION  -  Assembly  Area  West 


DAY  1:  TUESDAY,  SEPTEMBER  26,  2000 


7:00  AM  —   5:00  PM     REGISTRATION  -  Assembly  Area  West 

7:00  AM  —   9:00  AM     CONTINENTAL  BREAKFAST  -  Outside  Meramec  Room 


7:30  AM  —   9:00  AM 


9:00  AM  —  10:30  AM 


QUALITY:  THE  BASICS  (Optional)  -  Jefferson  B  Room 

Moderator:  Ann  Page,  Health  Care  Financing  Administration  (HCFA) 
Lawrence  Kleinman,  MD,  MPH,  Lehigh  Valley  Hospital 

PLENARY  SESSION  -  Meramec  Room 

Moderator:  Deirdre  Duzor,  HCFA 

WELCOME 

Eric  Goplerud,  Substance  Abuse  and  Mental  Health  Services  Administration 
(SAMHSA) 

Tom  Lenz,  HCFA  -  Region  VII 

KEYNOTE  ADDRESS  -  QUALITY:  PUTTING  IT  INTO 
PERSPECTIVE 

Rachel  Block,  HCFA 

COMBINING  MEASUREMENT  TOOLS  FOR  AN  EFFECTIVE  STATE 
STRATEGY 

Emma  Hoo,  Pacific  Business  Group  on  Health 


10:30  AM  —  10:45  AM     BREAK  -  Outside  Meramec  Room 

10:45  AM 


12:00  PM 


12:00  PM 


1:15  PM 


1:15  PM 


3:45  PM 


PLENARY  SESSION  -  Meramec  Room 
REPORTING  TO  YOUR  STAKEHOLDERS 

Moderator:  Kathleen  Blume,  HCFA 

Jeanne  McGee,  Ph.D,  McGee  and  Evers  Consulting,  Inc. 

LUNCH:  SPEAKER  -  QUALITY  OR  ELSE:  THE  FUTURE  OF 
AMERICAN  HEALTH  CARE  -  Missouri  Room 

Moderator:  Deirdre  Duzor,  FICFA 
Michael  Millenson,  William  M.  Mercer,  Inc. 

PLENARY  SESSION  -  Meramec  Room 

SAFEGUARDING  QUALITY:  STRUCTURAL  AND  OPERATIONAL 
STANDARDS 

Moderator:  Ann  Page,  HCFA 

Susan  Moran,  RN,  Michigan  Department  of  Community  Health 
Karen  Peed,  Minnesota  Department  of  Human  Services 
William  Tulloch,  MA,  National  Committee  for  Quality  Assurance 
Linda  Oliver,  RN,  PAHM,  Delman>a  Foundation  for  Medical  Care,  Inc. 
Julie  Tyler,  RN,  ABQAURP,  Delmarva  Foundation  for  Medical  Care,  Inc. 


DAY  1 :  TUESDAY,  SEPTEMBER  26,  2000  (Continued) 


3:45  PM  —   4:00  PM      BREAK  -  Outside  Meramec  Room 
4:00  PM  —   5:30  PM      BREAKOUT  SESSIONS 

ENCOUNTER  DATA: 

THE  BASICS  -  Jefferson  A  Room 

Moderator:  Sharon  Patterson,  HCFA 

Anthony  Asciutto,  MA,  MPH,  The  MEDSTAT  Group 

Marjorie  Porell,  MS,  Massachusetts  Division  of  Medical  Assistance 

LESSONS  LEARNED  WHEN  USING  ENCOUNTER  DATA  FOR 
MONITORING  QUALITY  IN  BEHAVIORAL  HEALTH  CARE  - 
Meramec  Room 

Moderator:  Elise  Young,  SAMHSA 

Jim  Linnane,  MPA,  Connecticut  Department  of  Social  Services 
Jack  Wackwitz,  Ph.D.,  Colorado  Mental  Health  Services 

LESSONS  LEARNED  WHEN  USING  ENCOUNTER  DATA  FOR 
MONITORING  QUALITY  IN  ACUTE/PREVENTIVE  CARE  - 
Jefferson  B  Room 

Moderator:  Carol  Crimi,  HCFA 

Joan  Kapowich,  RN,  Oregon  Department  of  Human  Services 
Patrick  Roohan,  MS,  New  York  Department  of  Health 


DAY  2:  WEDNESDAY,  SEPTEMBER  27,  2000 


7:30  AM  —   5:00  PM      REGISTRATION  -  Assembly  Area  West 

7:30  AM  —   8:30  AM      CONTINENTAL  BREAKFAST  -  Missouri  Room 


8:30  AM  —  10:00  AM     BREAKOUT  SESSIONS 

TAKING  THE  PULSE  OF  CONSUMERS  AND  PROVIDERS 
THROUGH  SURVEYS  -  Jefferson  B  &  C  Rooms 

Moderator:  Ann  Page,  HCFA 

Vasudha  Narayanan,  MS,  MBA,  MA,  The  WESTAT  Group 
Jeanne  McGee,  Ph.D.,  McGee  and  Evers  Consulting,  Inc. 

APPROACHES  AND  USES  OF  TOOLS  TO  IDENTIFY 
INDIVIDUALS  WITH  SPECIAL  HEALTH  CARE  NEEDS  - 
Jefferson  E  &  F  Rooms 

Moderator:  Deborah  Ashman,  HCFA 

Christina  Bethell,  Ph.D.,  MBA,  MPH,  Foundation  for  Accountability 
John  Neff,  MD,  Center  for  Children  with  Special  Needs  and  Chronic  Health 
Care  Conditions,  Children's  Hospital  and  Regional  Medical  Center 

1 0:00  AM  —  1 0: 1 5  AM     BREAK  -  Assembly  Area  West 

10:15AM  —11:15AM     BREAKOUT  SESSIONS  CONTINUE  -  Jefferson  B  &  C, 

Jefferson  E  &.  F  Rooms 


1  1:15  AM  —  12:30  PM      LUNCH  {On your  own) 


DAY  2:  WEDNESDAY,  SEPTEMBER  27,  2000  (Continued) 


12:30  PM  —  1:30  PM      PLENARY  SESSION 


USING  PERFORMANCE  MEASURES  -  Mississippi  Room 

Moderator:  Ann  Page,  HCFA 

Monica  Wassell,  RN,  MPP,  FMAS/DynCorp 

Michael  Bailit,  Bailit  Health  Purchasing,  LLC 


1:30  PM  —  3:00  PM      BREAKOUT  SESSIONS 


PERFORMANCE  MEASURES: 
BEHAVIORAL  HEALTH  -  Mississippi  Room 

Moderator:  Eric  Goplerud,  SAMHSA 
Barry  Kast,  MSW,  Oregon  Mental  Health  Division 
Deborah  Garnick,  Sc.D,  Brandeis  University 
Candice  Nardini,  MPH,  Value  Options 

ACUTE/PREVENTIVE  HEALTH  -  Jefferson  E  Room 

Moderator:  Deirdre  Duzor,  HCFA 

Mary  Fermazin,  MD,  MPA,  Health  Services  Advisory  Group 

Foster  Gesten,  MD,  New  York  Department  of  Health 

Alice  Lind,  RN,  MPH,  Washington  Medical  Assistance  Administration 

SPECIAL  NEEDS  POPULATIONS  -  Jefferson  F  Room 

Moderator:  Ann  Page,  HCFA 

Susan  Moran,  RN,  Michigan  Department  of  Community  Health 
Elaine  Haemmerle,  RN,  Arizona  Health  Care  Cost  Containment  System 


3:00  PM—  3:15  PM  BREAK  -  Assembly  Area  West 
3:15  PM—  5:30  PM      PLENARY  SESSION 


PERFORMANCE  IMPROVEMENT  PROJECTS  -  Mississippi  Room 

Moderator:  Sharon  Gilles,  HCFA 
Ruth  Martin,  MBA,  MPH,  Consultant 

Beth  Tortolani,  RN,  BS,  MS,  Massachusetts  Division  of  Medical  Assistance 
Diane  Herr,  Maryland  Health  Choice  and  Acute  Care  Administration 


DAY  3:  THURSDAY,  SEPTEMBER  28,  2000  (Closed  -  Participating  States  Only) 


7:30  AM  —  8:30  AM     REGISTRATION  AND  STORYBOARD  SET-UP  FOR  IHI 

COLLABORATIVE  -  Assembly  Area  West 

7:30  AM  —   8:30  AM     CONTINENTAL  BREAKFAST  -  Assembly  Area  West 

8:30  AM  —  10:30  AM     PLENARY  SESSION  -  Mississippi  Room 

INSTITUTE  FOR  HEALTHCARE  IMPROVEMENT:  ASTHMA  AND 
DEPRESSION  COLLABORATIVE 

Charles  Homer,  MD,  Institute  for  Healthcare  Improvement 
David  Katzelnick,  MD,  Madison  Institute  of  Medicine.  Inc. 
Carol  Haraden,  Ph.D.,  Institute  for  Healthcare  Improvement 


10:30  AM  —  10:45  AM 


BREAK  -  Assembly  Area  West 
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PAY  3:  THURSDAY,  SEPTEMBER  28,  2000  (Continued) 


10:45  AM  —  12:00  PM     INSTITUTE  FOR  HEALTHCARE  IMPROVEMENT  (Continued) 

Mississippi  Room 

12:00  PM  —    1:15  PM     LUNCH  {On your  own) 

1:15  PM—  2:45  PM     BREAKOUT  SESSIONS 

IMPROVING  CARE  FOR  BENEFICIARIES  WITH  ASTHMA  - 
Jefferson  E  Room 

Charles  Homer,  MD,  Institute  for  Healthcare  Improvement 

IMPROVING  CARE  FOR  BENEFICIARIES  WITH  DEPRESSION 
Jefferson  D  Room 

David  Katzelnick,  MD,  Madison  Institute  of  Medicine,  Inc. 


2:45  PM  —  3:00  PM 
3:00  PM   —  4:30  PM 


BREAK 


INSTITUTE  FOR  HEALTHCARE  IMPROVEMENT  (Continued) 
Mississippi  Room 
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SESSION  SUMMARIES 


SESSION  SUMMARIES 


SESSION  TITLE 

PRESENTER(S) 

DESCRIPTION 

DAY  1:  TUESDAY,  SEPTEMBER  26,  2000 

Quality:  The  Basics 

Larry  Kleinman 

This  optional  session  will  review  the  essentials  of  quality 
measurement:  defining  quality,  measuring  quality,  and  improving 
quality  of  care. 

Opening  Plenary: 

Quality:  Putting  it  into  Perspective 

Combining  Measurement  Tools 
for  an  Effective  State  Strategy 

Rachel  Block 
Emma  Hoo 

This  Keynote  session  will  address  the  importance  of  quality 
improvement.  The  session  will  provide  information  on  the 
importance  of  a  State  quality  strategy  and  the  approaches  States 
can  use  to  implement  a  quality  strategy.  The  session  will  also 
provide  information  on  the  resources  and  commitments  needed  by 
States  in  order  to  implement  a  quality  strategy. 

The  session  continues  with  an  illustration  on  how  the  Pacific 
Business  Group  on  Health  has  used  tools  in  a  coordinated  effort  for 
quality  improvement  purposes,  the  lessons  learned  from  these 
activities,  and  how  this  can  apply  to  State  Medicaid  agencies. 

Reporting  to  your  Stakeholders 

Jeanne  McCee 

This  plenary  session  will  provide  information  that  will  be  useful  to 
States  in  designing  reports  on  quality  of  health  care  provided  by 
Medicaid  MCOs  for  various  audiences  (e.g.  Medicaid  beneficiaries 
and  providers) 

Quality  or  Else:  The  Future  of 
American  Health  Care 

Michael  Millenson 

Michael  Millenson  is  the  author  of  Demanding  Medical  Excellence 
and  a  three-time  Pulitzer  Prize  nominee  as  a  reported  for  the 
Chicago  Tribune.  This  lunch  presentation  will  address,  from  a 
private  sector  perspective,  factors  that  influence  the  measurement 
and  improvement  of  health  care. 

Safeguarding  Quality:  Structural 
and  Operational  Standards 

Susan  Moran 
Karen  Peed 
William  Tulloch 
Linda  Oliver 
Julie  Tyler 

This  plenary  session  will  identify  how  to  monitor  compliance  with 
structural  and  operational  standards  as  part  of  a  State  quality 
strategy  from  three  different  perspectives  -  a  State  Health 
Department,  an  Accrediting  Organization,  and  an  External  Quality 
Review  Organization. 

Encounter  Data:  The  Basics 

Anthony  Asciutto 
Marjorie  Porell 

This  breakout  session  will  focus  on  the  fundamental  issues  involved 
in  the  collection  of  encounter  data.  Issues  to  be  discussed  include 
how  to  collect  and  validate  encounter  data,  the  advantages  and 
limitations  of  encounter  data  as  a  data  source,  accuracy  and 
completeness,  comprehensiveness  (i.e.,  all  variables  required),  and 
the  importance  of  leadership  and  commitment  of  resources  over 
time. 

Encounter  Data: 

Lessons  Learned  when  Using 
Encounter  Data  for  Monitoring 
Quality  in  Behavioral  Health  Care 

Jim  Linnane 
Jack  Wackwitz 

This  breakout  session  will  identify  how  to  collect  and  use  valid  and 
reliable  data  for  quality  measurement  purposes.  The  session  will 
focus  on  the  strengths  and  limitations  of  using  encounter  data  to 
measure,  assess,  and  improve  behavioral  health  care. 

Encounter  Data: 

Lessons  Learned  when  Using 
Encounter  Data  for  Monitoring 
Quality  in  Acute/Preventive  Care 

Joan  Kapowich 
Patrick  Roohan 

This  breakout  session  will  identify  how  to  collect  and  use  valid  and 
reliable  data  for  quality  measurement  purposes.  The  session  will 
focus  on  the  strengths  and  limitations  of  using  encounter  data  to 
measure,  assess,  and  improve  acute/preventive  care. 

SESSION  TITLE 

PRESENTER(S) 

DESCRIPTION 

DAY  2:  WEDNESDAY,  September  27,  2000 

Taking  the  Pulse  of  Consumers 
and  Providers  Through  Surveys 

Vasudha  Narayanan 
Jeanne  McCee 

This  breakout  session  will  provide  information  that  will  be  useful 
to  States  in  designing  and  using  different  types  of  surveys  of 
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care.  This  session  includes  an  overview  of  the  EQR  Survey 
Protocol,  a  discussion  of  consumer  surveys  being  used  by  States, 
and  a  discussion  of  the  strengths  and  weaknesses  of  provider 
surveys. 

Approaches  and  Uses  of  Tools  to 
Identify  Individuals  with  Special 
Health  Care  Needs 

Christina  Bethell 
John  Neff 

This  breakout  session  will  provide  information  on  different 
approaches  and  uses  of  screening  tools  to  identify  individuals  with 

DL/CWldl  IICdlLII  Lul  C  1  IcCUo. 

Using  Performance  Measures 

Michael  Bailit 
Monica  Wassell 

This  plenary  session  will  address  the  BBA  regulation  with  respect 
to  performance  measures  and  will  provide  a  brief  overview  of  the 
EQR  protocol  for  validating  performance  measures.  The  session 
will  also  address  the  fundamentals  of  performance  measurement 
such  as  its  relationship  to  other  quality  improvement  approaches, 
the  origins  of  performance  measurement  in  the  health  care 
industry,  and  commonly  used  performance  measures. 

Pprfnrmanrp  Mpasiirp^* 

Behavioral  Health 

Barry  Kast 
Deborah  Camick 
Candice  Nardini 

This  breakout  spssion  will  discuss  the  use  of  behavioral  hpalth 
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performance  measures.  The  presenters  will  discuss  their 
experiences  using  performance  measures  and  will  address  the 
following:  the  origin  of  their  measures,  problems  encountered 
when  using  performance  measures,  the  pros  and  cons  of  using 
standardized  versus  State-developed  measures,  the  need  for 
validation  of  performance  measures,  issues  in  using  performance 
measures  once  produced,  and  successes  experienced  by  States  in 
improving  quality  through  the  use  of  performance  measures. 

Acute/Preventive 
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Foster  Cesten 
Alice  Lind 

i  ms  uicaKouc  session  win  discuss  enc  use  or  acute/ preventive 
health  performance  measures.  The  presenters  will  discuss  their 
experiences  using  performance  measures  and  will  address  the 
following:  the  origin  of  their  measures,  problems  encountered 
when  using  performance  measures,  the  pros  and  cons  of  using 
standardized  versus  State-developed  measures,  the  need  for 
validation  of  performance  measures,  issues  in  using  performance 
measures  once  produced,  and  successes  experienced  by  States  in 
improving  quality  through  the  use  of  performance  measures. 

Performance  Measures: 

Special  Needs  Population 

Susan  Moran 
Elaine  Haemmerle 

This  breakout  session  will  discuss  the  use  of  performance 
measures  for  beneficiaries  with  special  health  care  needs.  The 
presenters  will  discuss  their  experiences  using  performance 
measures  and  will  address  the  following:  the  origin  of  their 
measures,  problems  encountered  when  using  performance 
measures,  the  pros  and  cons  of  using  standardized  versus  State- 
developed  measures,  the  need  for  validation  of  performance 
measures,  issues  in  using  performance  measures  once  produced, 
and  successes  experienced  by  States  in  improving  quality  through 
the  use  of  performance  measures. 

Performance  Improvement 
Projects 

Ruth  Martin 
Beth  Tortolani 
Diane  Herr 

This  plenary  session  will  address  how  to  conduct  and  validate 
performance  improvement  projects.  An  overview  of  the  EQR 
protocol  will  be  provided.  In  addition,  examples  will  be  provided 
from  two  State  Medicaid  agencies  on  their  experiences 
conducting  these  projects. 
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RACHEL  BLOCK 


Rachel  Block  is  the  Deputy  Director  of  the  Center  for  Medicaid  and  State 
Operations,  Health  Care  Financing  Administration.  In  conjunction  with  the 
Center's  Director,  she  provides  leadership  in  the  development  and 
implementation  of  national  policies  governing  Medicaid,  survey  and  certification, 
the  Children's  Health  Insurance  Program  and  insurance  reform  provisions  of  the 
Health  Insurance  Portability  and  Accountability  Act  (HIPAA).  CMSO  also  serves 
as  the  focal  point  for  all  HCFA  interactions  with  States,  local  governments  and 
Native  American  and  Alaskan  Native  Tribes. 

Since  joining  HCFA  in  1994,  Rachel  has  also  served  as:  Director  of  CMSO's 
Data  and  Systems  Group  where  she  was  responsible  for  the  development  and 
support  of  systems  for  Medicaid  and  survey  and  certification,  as  well  as 
information  analysis  and  technical  assistance  for  internal  and  external 
customers;  Director  of  the  Medicaid  Managed  Care  Team  where  she  was 
responsible  for  overseeing  the  review  and  approval  of  Section  1915(b)  Medicaid 
managed  care  waivers,  and  provided  leadership  on  major  policy  and  operational 
issues  relating  to  Medicaid  managed  care;  and  as  Director  of  the  Office  of 
Planning  and  Special  Initiatives,  and  Associate  Director  of  HCFA's  Medicaid 
Bureau. 

Before  she  joined  HCFA,  Rachel  served  as  Special  Adviser  on  health  care  to 
Vermont's  Governor  Howard  Dean,  MD  and  Executive  Director  of  the  Vermont 
Health  Care  Authority;  Executive  Director  for  the  Majority  Leader  of  the  New  York 
State  Assembly,  with  principal  responsibility  for  health  and  social  services  policy 
(1988-1992),  Director  of  the  Subcommittee  on  Health  Insurance,  New  York  State 
Council  on  Health  Care  Financing  (1985-1988),  and  in  various  staff  positions 
with  the  New  York  State  Assembly's  Committee  on  Health  (1983-1985)  and 
Higher  Education  (1979-1982). 
Phone:        410  786-3230 
Fax:  410  786-0025 

E-Mail:  RBIock@HCFA.GOV 


Combining  Measurement  Tools 
for  an  Effective  State  Strategy 

Emma  Hoo 
Pacific  Business  Group  on  Health 
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Emma  Hoo 


Emma  Hoo  is  the  Associate  Director  of  Value-Based  Purchasing  at  the  Pacific  Business 
Group  on  Health  (PBGH).  She  manages  the  annual  negotiations  for  20  large  businesses 
in  the  PBGH  Negotiating  Alliance,  representing  over  $400  million  in  annual  premiums. 
In  addition,  she  is  responsible  for  the  operations  of  Pacific  Health  Advantage,  a 
purchasing  pool  of  9,000  small  businesses  and  140,000  members,  also  known  as  the 
Health  Insurance  Plan  of  California  (HIPC).  The  Negotiating  Alliance  has  successfully 
negotiated  with  commercial  and  Medicare  California  HMO's  on  rates,  benefits,  and 
performance  measures  since  1994. 

Previously,  Ms.  Hoo  was  Director  of  Operations  for  a  Northern  California  medical  group 
where  she  is  responsible  for  contracting,  data  analysis,  and  information  systems.  Ms.  Hoo 
received  a  B.S.  in  Social  Studies  from  Harvard  University. 
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Reporting  to  Your  Stakeholders 

Jeanne  McGee 
McGee  and  Evers  Consulting  Services,  Inc. 


# 


m 


Jeanne  McGee,  PhD 

McGee  &  Evers  Consulting,  Inc. 


voice 


360  -  574  -  4744 


1924  NW  Tilth  Street 


fax: 


360  -  574  -  4840 


Vancouver,  Washington  98685 


email: 


jmcgee@pacifier.com 


Jeanne  McGee  is  a  health  care  consultant  in  Vancouver,  Washington.  A  sociologist  with  20  years 
of  experience,  she  has  taught  at  Duke  University,  Duke  Medical  Center,  and  the  University  of 
Oregon.  For  more  than  ten  years  her  consulting  firm  has  done  custom  research  and  evaluation  for 
clients  that  include  government  agencies,  health  plans,  employers,  hospitals,  foundations,  and 
research  institutes.  Much  of  her  recent  work  involves  health  care  quality:  designing  and 
conducting  surveys  to  get  consumers'  and  providers'  reactions  to  health  plans  and  medical  care; 
designing  and  testing  ways  to  share  comparative  health  care  information  with  consumers  and 
purchasers;  and  helping  providers  use  information  for  internal  quality  improvement.  Last  year  Dr. 
McGee  wrote  a  detailed  technical  assistance  manual  for  HCFA  called  Writing  and  Designing 
Print  Materials  for  Beneficiaries:  A  Guide  for  State  Medicaid  Agencies.  Working  with  Arizona's 
Medicaid  program,  she  has  recently  completed  a  monograph  about  conducting  surveys  of 
physicians  and  dentists  to  get  their  evaluations  of  Medicaid  managed  care  programs  and  health 
plans.  She  is  presently  working  with  Medicaid  programs  in  Oregon  and  Washington  to  explore 
the  use  of  CAHPS  survey  questions  and  report  formats  with  beneficiaries  who  speak  Spanish, 
Russian,  and  Vietnamese. 
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Quality  or  Else:  The  Future  of 
American  Health  Care 

Michael  L.  Millenson 
William  M.  Mercer,  Inc. 
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Michael  L.  Millenson 


Michael  L.  Millenson,  a  principal  in  the  Health  Care  and  Group  Benefits  practice  of 
William  M.  Mercer,  Incorporated,  is  the  author  of  Demanding  Medical  Excellence: 
Doctors  and  Accountability  in  the  Information  Age  (University  of  Chicago  Press).  The 
book,  which  is  the  first  examination  of  quality-of-care  issues  for  the  general  public, 
recently  appeared  in  paperback  with  an  updated  afterword. 

National  Public  Radio  called  Millenson  "in  the  vanguard  of  the  movement"  to  measure 
and  improve  the  quality  of  medical  care.  Meanwhile,  former  surgeon  general  C.  Everett 
Koop  said  that  Demanding  Medical  Excellence  "  should  be  read  by  everyone  who  has  a 
stake  in  the  quality  of  American  medicine  —  patients,  physicians  and  policymakers". 

Before  joining  Mercer,  Millenson  was  a  reporter  for  the  Chicago  Tribune,  where  he  was 
nominated  for  a  Pulitzer  Prize  three  times. 

A  sought-after  speaker,  Millenson  has  addressed  numerous  national  organizations, 
testified  before  Congress  and  been  quoted  extensively  in  the  news  media.  His  articles 
have  appeared  in  publications  such  as  Health  Affairs,  World  Book  Encyclopedia  and 
Medicine  &  Health.  The  research  for  his  book  was  funded  by  an  Investigator  Award  in 
Health  Policy  Research  from  the  Robert  Wood  Johnson  Foundation. 

Millenson  joined  Mercer  in  its  Chicago  office  in  1996.  He  is  also  a  visiting  scholar  at 
Northwestern  University's  Institute  for  Health  Services  Research  and  Policy  Studies. 
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Safeguarding  Quality:  Structural 
and  Operational  Standards 

Susan  Moran 
Michigan  Department  of  Community  Health 


Susan  Moran 


Sue  Moran  is  Manager  of  the  Quality  System  Section  within  the  Quality 
Improvement  and  Customer  Service  Bureau  of  Michigan's  Medical  Services 
Administration  (the  State's  Medicaid  agency).  Her  section  is  responsible  for 
many  of  the  Medicaid  program  quality  oversight  initiatives,  including  the  external 
quality  review,  HEDIS  analysis,  consumer  satisfaction  survey,  consumer  guide, 
and  encounter  data  reporting.  Sue  has  been  employed  with  the  Michigan 
Department  of  Community  Health  since  September  1998.  Prior  to  joining  DCH, 
she  worked  for  over  ten  years  in  the  managed  care  industry,  primarily  in  quality 
assessment  and  improvement.  Sue's  background  is  in  nursing  and  public  health. 

She  will  discuss  with  us  the  work  Michigan  has  done  in  the  development  and 
implementation  of  a  structured  site  visit  tool  modeled  after  their  Medicaid 
contract  and  used  by  the  state  to  monitor  compliance  with  structural  and 
operational  standards. 
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and  Operational  Standards 

Karen  Peed 
Minnesota  Department  of  Human  Services 
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Karen  Peed 


Karen  Peed  has  been  with  the  Minnesota  Department  of  Human  Services  since  1986.  She  has  held 
positions  relating  to  long  term  care  and  community  based  services,  as  well  as  health  care.  She  is 
currently  the  Contract  Management  and  Development  Supervisor  in  the  Minnesota  Department  of 
Human  Services  Health  Care  Purchasing  and  Service  Delivery  Division.  Minnesota  currently  enrolls 
almost  325,000  people  in  managed  care  plans. 
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Bill  is  a  Senior  Accreditation  Manager  who  has  been  with  NCQA  since 

1997,  participating  in  and  coordinating  surveys  for  the  MCO,  CVO,  and 
MBHO  programs.  In  addition  to  his  survey  work,  Bill  has  made  educational 
presentations  for  NCQA,  and  was  one  of  the  leaders  of  the  team  that 
adapted  the  MCO  DCT/report  template  and  electronic  roadmaps  for  the 

1998,  1999,  and  2000  survey  years. 

Bill  attended  Williams  College  for  his  undergraduate  work,  and  has  a 
Master's  in  Economics  from  the  University  of  Maryland.  For  seven  years 
prior  to  joining  NCQA,  Bill  worked  with  a  health  care  consulting  firm,  HTA 
(now  Covance),  in  Washington,  DC,  helping  medical  product  manufacturers 
work  in  the  managed  care  setting.  During  his  undergraduate  years,  Bill  co- 
authored  two  papers  on  the  psychology  of  juror  decision-making,  based  on 
original  research. 
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University  and  will  complete  her  degree  in  2001. 
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responsible  for  operations  in  the  corporate  and  regional  offices  in  five  states  and  acts  as 
liaison  with  Federal  and  State  contractors. 


Linda  Oliver,  R.N.,  PAHM 
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Ms.  Oliver  is  currently  Managed  Care  Manager  with  Delmarva  Foundation  for  Medical 
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VALIDATING  MCO  ENCOUNTER  DATA 


I.  PURPOSE  OF  THE  PROTOCOL 

Encounter  data  (i.e.,  data  on  the  distinct  health  care  services  provided  to  each  Medicaid  managed 
care  enrollee)  can  be  a  useful  source  of  information  for  States,  as  well  as  managed  care 
organizations  (MCOs)  and  Prepaid  Health  Plans  (PHPs).  Encounter  data  can  be  used  to  assess 
and  improve  quality,  as  well  as  monitor  program  integrity  and  determine  capitation  payment 
rates.  However,  in  order  for  encounter  data  to  effectively  serve  these  purposes,  it  must  be  valid; 
i.e.,  complete  and  accurate.  At  present,  completeness  and  accuracy  of  encounter  data  vary  across 
States,  MCOs,  and  PHPs.  This  protocol  specifies  processes  for  assessing  the  completeness  and 
accuracy  of  encounter  data  submitted  by  MCOs  and  PHPs  to  the  State.  It  also  can  assist  in  the 
improvement  of  the  processes  associated  with  the  collection  and  submission  of  encounter  data  to 
State  Medicaid  agencies. 

II.  ORIGIN  OF  THE  PROTOCOL 

This  protocol  was  developed  from  documents  in  both  the  public  and  private  sectors,  as  well  as 
interviews  with  personnel  from  three  State  Medicaid  agencies  (Alabama,  Arizona  and  Oregon) 
experienced  in  the  collection  of  encounter  data.  The  documents  reviewed  included:  1)  A  Guide 
for  States  to  Assist  in  the  Collection  and  Analysis  of  Managed  Care  Data  -  second  edition1 
(draft);  2)The  MEDSTAT  Group  (MEDSTAT)'s  Final  Design  Report  for  Verification  of 
Encounter  Data  (part  of  the  evaluation  of  the  Medicare  Choices  Demonstration);  and  3)  the 
National  Committee  for  Quality  Assurance  (NCQA)'s  1999  HEDIS®  publication:  Volume  5, 
HEDIS  Compliance  Audit™  Standards  and  Guidelines. 

Beginning  in  1995,  HCFA  and  MEDSTAT  began  developing  a  series  of  tools  to  help  State 
Medicaid  agencies  collect,  validate  and  use  encounter  data  for  managed  care  program 
management  and  oversight.  The  tools  and  approaches  developed  for  this  contract  were  further 
refined  and  narrowed  as  part  of  the  pseudo-claims  (encounter  data)  validation  project  HCFA 
commissioned  for  the  Medicare  Choices  Demonstration.  For  that  project,  HCFA  specifically 
requested  the  development  and  application  of  a  statistically  reliable  encounter  data  validation 
process  using  medical  records  as  the  reference  information.  MEDSTAT  has  used  similar 
approaches  for  validating  Medicaid  managed  care  encounter  data  in  a  number  of  States,  although 
these  approaches  have  varied  depending  on  the  sophistication  of  the  MCO/PHP  and  State 
information  systems,  the  amount  of  encounter  data  collected,  and  each  State's  approach  to 
improving  the  quality  of  encounter  data.  However,  in  all  of  these  States,  MEDSTAT  has  used  an 
approach  composed  of  three  core  activities: 


The  final  version  of  this  document  is  available  online  at:  (www.hcfa.gov/medicaid/enguide2.htm) 
HCFA/CMSO/QPMG/DQSM:  August  13,  2000  DRAFT  Page  1 


( 1 )  Assessment  of  the  MCO  or  PHP's  information  system  (IS); 

(2)  Analysis  of  MCO  or  PHP  electronic  encounter  data  for  accuracy  and 
completeness,  including  analysis  of  data  reasonableness;  and 

(3)  Review  of  medical  records  for  additional  confirmation  of  findings. 

As  part  of  the  validation  process,  MEDSTAT  has  developed  data  collection  instruments  designed 
to  evaluate,  troubleshoot  and  facilitate  improvement  of  encounter  data  and  the  information 
systems  from  which  encounter  data  are  produced.  These  instruments  are  intended  to  be 
completed  by  the  MCO/PHP.  The  information  obtained  through  these  instruments  is  to  be 
confirmed  through  face-to-face  interviews  of  MCO/PHP  staff  by  staff  conducting  the  assessment 
of  the  MCO/PHP  IS.  The  tools  also  are  designed  to  be  sensitive  to  the  burden  placed  on  an 
MCO/PHP  by  information  collection,  while  maintaining  the  integrity  of  the  information 
collected. 

The  NCQA  HEDIS  Compliance  Audit  tool  was  designed  for  auditing  encounter  data  when 
encounter  data  are  used  to  calculate  certain  performance  measures.  It  also  includes  an  IS 
assessment  analogous  to  the  first  of  the  three  elements  of  the  MEDSTAT  validation  process.  The 
NCQA  tool  is  designed  to  gather  information  about  MCOs'/PHPs'  IS  capabilities  across  all 
payors,  rather  than  concentrating  specifically  on  Medicaid  or  Medicare.  The  MEDSTAT  tools 
were  developed  for  Medicaid  and  Medicare  only,  although  MEDSTAT  also  has  companion 
approaches  designed  for  commercial  payors. 

All  the  validation  processes  reviewed  address  the  collection  of  information  about  the 
MCO's/PHP's  IS  capabilities  as  a  first  step.  All  also  include  medical  record  review  as  a 
component  of  the  validation  method.  Each  document  provides  a  method  to  calculate  a 
statistically  valid  sample  size  for  the  medical  record  review.  Interviews  with  State  Medicaid 
agency  personnel  found  that  their  protocols  also  include  medical  record  review. 

The  NCQA  and  MEDSTAT  tools  also  are  similar  in  that  they:  1)  contain  pre-onsite  visit 
questionnaires  to  be  completed  by  MCO/PHP  staff,  and  site  visit  interview  forms  to  be 
completed  by  staff  performing  the  IS  assessment;  2)  require  MCOs/PHPs  to  provide  information 
on  the  level  of  specificity  of  the  diagnosis  and  procedure  coding  systems  used;  3)  devote 
considerable  attention  to  medical  record  review;  and  4)  explore  the  issue  of  provider  contracts 
and  physician  compensation.  Provider  contracts  and  compensation  are  significant  elements  in 
understanding  the  flow  of  encounter  data  from  the  providers  to  the  MCOs/PHPs,  which 
influences  the  timeliness  and  completeness  of  encounter  data. 

III.  PROTOCOL  OVERVIEW 

This  protocol  is  based  almost  entirely  on  the  guide  for  States  developed  by  MEDSTAT  for 
validation  of  encounter  data.  The  elements  contained  in  MEDSTAT's  document  are  consistent 
with  the  other  documents  reviewed. 
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This  protocol  also  makes  the  following  assumptions: 

1 .  For  the  purposes  of  this  protocol,  an  encounter  refers  to  the  electronic  record  of  a  service 
provided  to  an  MCO/PHP  enrollee  by  both  institutional  and  practitioner  providers 
(regardless  of  how  the  provider  was  paid)  when  the  service  would  traditionally  be  a 
billable  service  under  fee-for-service  (FFS)  reimbursement  systems.  Encounter  data 
provides  substantially  the  same  type  of  information  that  is  found  on  a  claim  form  (e.g., 
UB-92  or  HCFA  1500),  but  not  necessarily  in  the  same  format. 

2.  The  State  will  further  specify  an  operational  definition  of  an  "encounter"  and  the  types  of 
encounters  (e.g.,  physician,  hospital,  dental,  vision,  laboratory  etc)  for  which  encounter 
data  are  to  be  provided.  The  State  will  also  specify  the  information  (data  fields)  to  be 
submitted  for  each  type  of  encounter. 

3.  Encounter  data  can  be  considered  "complete"  when  they  can  be  used  to  describe  the 
majority  of  services  that  have  been  provided  to  Medicaid  beneficiaries  who  are  enrollees 
of  a  MCO/PHP.2 

4.  Development  of  accurate  and  complete  encounter  data  is  an  iterative  process.  Because 
encounter  data  are  an  outgrowth  of  MCO/PHP  IS  and  data  policies,  it  is  often  not  possible 
for  MCOs  and  PHPs  to  overcome  all  limitations  in  their  IS  and  data  policies  in  one  year. 
As  a  result,  in  the  first  year  that  a  State  requires  the  submission  of  encounter  data  from  its 
MCOs  and  PHPs,  the  data  may  be  significantly  incomplete  and  contain  errors.  Improving 
the  completeness  and  accuracy  will  take  place  through  continuous  quality  improvement 
(CQI)  processes  implemented  year  after  year.  Because  of  this,  States  will  need  to  develop 
a  "phased-in"  approach  for  using  standards  for  encounter  data  accuracy  and 
completeness.  "Phased-in"  standards  acknowledge  the  start-up  issues  affecting  both 
MCO/PHPs  and  State  Medicaid  information  systems  receiving  the  encounter  data. 

5.  The  State  will  establish  standards  for  encounter  data  accuracy  and  completeness. 

6.  States  will  specify  objective  standards  to  which  encounter  data  submitted  by  their 
Medicaid  MCOs/PHPs  will  be  compared.  These  standards  can  be  national,  regional  or 
State  standards,  as  discussed  in  ACTIVITY  3  of  this  protocol. 

The  protocol  consists  of  five  sequential  activities: 

( 1 )      Review  of  State  requirements  for  collection  and  submission  of  encounter  data; 


'A  State  may  decide  to  use  other  sources  of  information,  such  as  an  immunization  registry,  to  substantiate  or 
complete  the  information  on  the  provision  of  services  to  Medicaid  beneficiaries  when  the  encounter  data  format  does 
not  easily  or  accurately  capture  the  required  information. 
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(2)  Review  of  each  MCO's/PHP's  capability  to  produce  accurate  and  complete 
encounter  data; 

(3)  Analysis  of  MCO/PHP  electronic  encounter  data  for  accuracy  and  completeness; 

(4)  Review  of  medical  records,  as  appropriate,  for  additional  confirmation  of 
findings;  and 

(5)  Submission  of  findings. 

IV.  PROTOCOL  ACTIVITIES 

ACTIVITY  1:         Review  State  requirements  for  encounter  data  collection  and 
submission. 

Prior  to  performing  encounter  data  validation,  the  External  Quality  Review  Organization 
(EQRO)3  needs  to  be  familiar  with  the  State's  requirements  for  the  collection,  processing  and 
submission  of  encounter  data  by  MCOs  and  PHPs  to  the  State.  Some  State  requirements  may  be 
unique  to  a  particular  State  program.  States  need  to  provide  the  EQRO  with:  1)  the  State's 
requirements  for  collection  and  submission  of  encounter  data  by  MCOs  /  PHPs.  (These  typically 
are  found  as  specifications  in  the  contracts  between  the  State  and  the  MCO  /  PHP.)  2)  the  data 
submission  format  specified  by  the  State  for  MCO/PHP  use;  3)  the  State's  data  dictionary;  4)  an 
explanation  of  the  information  flow  from  the  MCO/PHP  to  the  State;  5)  State  standards  for 
encounter  data  completeness  and  accuracy;  6)  the  time  frames  for  data  submission;  7)  any 
historical  problems  experienced  in  this  process;  and  8)  any  other  information  relevant  to 
encounter  data  validation. 

The  EQRO  should  also  obtain  from  the  State  a  listing  of  the  types  of  encounter  data  to  be 
validated.  For  each  type  of  encounter  data  (e.g.,  office  visit,  inpatient,  laboratory,  et  al.)  to  be 
validated,  the  State  should  specify  the  rates  of  missing,  surplus,  or  erroneous  encounters  (as 
defined  below)  that  it  will  find  acceptable.  The  Acceptable  Error  Rates  Specification  Form  below 
(or  a  similar  form)  can  be  used  to  summarize  these  specifications  for  each  of  the  different  types 
of  encounter  data. 

The  State  also  should  specify  acceptable  rates  of  accuracy  and  completeness  for  each  data  field 
submitted  for  each  encounter  type.  Attachment  1  contains  MEDSTAT's  recommendations  for 
eventual  accuracy  and  completeness  standards  for  typical  data  fields.  The  EQRO  will  need  to 
tailor  this  chart  or  generate  a  form  or  forms  similar  to  Attachment  1  to  identify  accuracy  and 
completeness  standards  specified  by  the  State  for  all  data  fields  the  State  requires  for  the  different 
types  of  encounters.  The  standards  should  be  more  lenient  in  the  early  years  of  collecting 


It  is  recognized  that  a  State  may  choose  an  organization  other  than  an  EQRO  as  defined  in  Federal 
regulation  to  perform  encounter  data  validation.  However,  for  convenience,  in  this  protocol  we  use  the  term 
"external  quality  review  organization"  (EQRO)  to  refer  to  any  organization  conducting  validation  of  encounter  data. 
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encounter  data,  and  more  stringent  as  MCO/PHP  IS  capabilities  improve  over  time. 


Acceptable  Error  Rates  Specification  Form 


iype  01  iMiLOUiii c r 

jtii  ror  1  yp*- 

A  ppDntonlo  17  i*i*ai*  »J  ofn 

/\LLcpi<tUit:  n*rrur  tvdlc 

Office  Visit  (excludes  dental  and  mental 
health  /  substance  abuse  visits) 

• 

Missing 

<  % 

Surplus 

<  % 

Erroneous 

<  % 

Office  Visit  -  mental  health  /  substance 
abuse 

Missing 

<  % 

Surplus 

<  % 

Erroneous 

<  % 

Office  Visit  -  dental 

Missing 

<  % 

Surplus 

<  % 

Erroneous 

<  % 

Inpatient  admission  -  (excludes  mental 
health  /  substance  abuse  visits) 

Missing 

<  % 

Surplus 

<  % 

Erroneous 

<  % 

Inpatient  admission  -  mental  health  / 
substance  abuse 

Missing 

<  % 

Surplus 

<  % 

Erroneous 

<  % 

Other  types  of  encounters  as  specified 
by  the  State;  e.g.,  laboratory,  pharmacy, 
physical  therapy. 

Missing 

<  % 

Surplus 

<  % 

Erroneous 

<  % 

The  EQRO  should  add  as  many 
additional  rows  to  this  chart  as  needed 
to  incorporate  all  types  of  encounters 
specified  by  the  State. 

Definitions: 


Missing  -  encounters  that  occurred  but  are  not  represented  by  an  electronic  record. 

Surplus  -  encounters  which  are  represented  by  an  electronic  record,  but  either  did  not  occur  or 

duplicated  other  records. 
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Erroneous  -  encounters  that  occurred  and  are  represented  by  an  electronic  record,  but  contain 
incorrect  data  elements. 

Acceptable  Error  Rate  -  the  maximum  percentage  of  missing,  surplus,  or  erroneous  records  that 
the  State  is  willing  to  consider  acceptable. 

ACTIVITY  2:         Review  each  MCO/PHP's  capability  to  produce  accurate  and 
complete  encounter  data. 

It  is  not  feasible  to  review  all  encounters  that  beneficiaries  have  with  MCO/PHP  providers  to 
assess  whether  they  are  completely  and  accurately  recorded.  Therefore,  efficiently  assessing 
encounter  data  completeness  and  accuracy  involves:  1)  determining  if  the  MCO/PHP  has 
structured  its  information  system  in  a  way  that  is  likely  to  capture  complete  and  accurate 
encounter  data;  and  then,  2)  examining  more  closely  the  data  produced  by  the  IS  system  to  detect 
patterns  that  can  indicate  its  completeness  and  accuracy.  Activity  2  addresses  the  first  of  these 
two;  the  second  is  addressed  in  Activity  3. 

Activity  2  attempts  to  answer  the  question:  "To  what  degree  is  an  MCO's  or  PHP's  information 
system  likely  to  produce  complete  and  accurate  information  on  all  encounters  between  Medicaid 
enrollees  and  their  providers  (both  institutions  and  practitioners).  Reviewing  the  capability  of 
each  MCO/PHP  to  do  so  is  accomplished  through  two  activities: 

1.  Reviewing  a  standardized  assessment  of  each  MCO/PHP's  IS  capabilities;  and 

2.  Interviewing  personnel  at  each  MCO/PHP  to  augment  information  obtained 
through  the  standardized  assessment. 

Step  1 :  Review  or  conduct  a  standardized  assessment  of  each  MCO/PHP's  IS  capabilities. 

A  standardized  assessment  (i.e.,  an  assessment  that  does  not  vary  by  the  individual  performing 
the  review  or  by  the  questions  being  asked)  is  necessary  to  promote  reliable  assessments  of 
information  systems.  A  standardized  tool  and  approach  for  conducting  such  a  review  is  found  in 
Appendix  A  (Information  System  Capabilities  Assessment  (ISCA)  for  Managed  Care 
Organizations  and  Prepaid  Health  Plans). 

An  MCO  or  PHP  may  already  have  undergone  such  an  assessment  of  its  IS.  For  example, 
assessment  of  IS  is  conducted  when  validating  performance  measures  and  performing 
accreditation  reviews.  The  EQRO  needs  to  determine  if  the  MCO/PHP  whose  encounter  data  are 
being  validated  has  already  undergone  such  a  review,  and  if  so,  if  the  review  findings  are  current. 
If  a  recent  IS  assessment  has  been  conducted,  the  EQRO  should  receive  a  copy  of  the  findings, 
review  the  results  of  the  prior  assessment,  and  seek  more  recent  information  where  necessary.  If 
the  MCO/PHP  has  not  recently  undergone  an  assessment,  one  will  need  to  be  conducted  as  part 
of  encounter  data  validation  consistent  with  the  process  described  in  Appendix  A. 
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Whether  the  EQRO  reviews  the  results  of  an  earlier  IS  assessment  or  conducts  its  own 
assessment,  the  content  included  in  Appendix  A  should  be  addressed.  This  content  and  the 
reasons  for  its  significance  include  the  following: 

■  General  Information 

1 .  Managed  Care  Model  Type:  Encounter  data  completeness  and  accuracy  are  likely 
to  be  better  in  staff  model  than  non-staff  model  MCOs/PHPs. 

2.  Year  of  incorporation:  Encounter  data  accuracy  and  completeness  is  likely  to  be 
better  in  more  mature  MCOs/PHPs. 

3.  Member  enrollment:  A  larger  enrollment  may  indicate  that  the  MCO/PHP  has 
more  experience  working  with  encounter  data,  but  may  also  offer  more 
opportunity  for  errors. 

■  Information  Systems  Capabilities 

1 .       System  descriptions:  This  provides  an  indication  of  the  MCO's/PHP's  overall 
level  of  data  management  sophistication. 

■  Data  Acquisition  Capabilities 

1 .  Forms  used:  If  the  MCO/PHP  is  not  using  standard  claims  forms  (e.g.,  UB  92  or 
HCFA  1500)  or  encounter  forms  which  are  similar  to  the  HCFA  1500  or  the  UB 
92,  the  forms  used  should  be  reviewed  to  ensure  that  they  capture  key  data 
elements. 

2.  Submission  methods:  Processing  paper  forms  is  more  prone  to  error  than  direct 
electronic  data  submission. 

3.  Required  data  fields  (data  elements):  Standard  measures  of  plan  performance 
typically  require  the  availability  of  data  on:  patient  date  of  birth/age,  sex,  place  of 
service,  diagnoses,  procedures,  dates  of  service,  revenue  codes,  and  provider 
specialty. 

4.  Number  of  diagnosis  and  procedure  codes  retained  in  data  fields:  Data  fields 
should  allow  a  minimum  of  two  diagnoses  and  two  procedure  codes  to  be 
retained. 

5.  Coding  schemes:  Knowledge  of  the  coding  schemes  used  by  the  MCO/PHP  is 
necessary  to  verify  the  accuracy  of  their  use. 

■  Claims/Encounter  Processing 

1 .  Processing  issues:  Points  in  the  process  where  errors  are  particularly  likely  to 
occur  should  be  identified. 

2.  Edit  checks:  MCOs/PHPs  should  have  an  established,  standard  set  of  edits  which 
verify  field  content  and  consistency. 
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■        Enrollment  Issues 


1 .  Type  and  frequency  of  updates:  Infrequent  or  inaccurate  updates  will  result  in 
invalid  encounter  data. 

2.  Use  of  unique  identifiers:  Without  reliable  identification  of  enrollees  and 
providers,  encounter  data  validity  is  not  possible. 

■  Vendor/Contractor  Data 

1 .  Data  submission  policies:  Consistently  applied  policies  that  leave  no  room  for 
variations  in  interpretation  increase  the  accuracy  of  the  data  submitted. 

2.  Contract  requirements:  Data  are  more  likely  to  be  complete  when  they  are 
required  as  a  condition  of  payment  than  when  they  are  optional. 

■  Provider  Contracting  Arrangements 

1 .  Compensation  arrangements:  Salaried  providers  will  submit  data  on  a  timely 
basis  if  data  submission  is  a  parameter  in  their  contract  with  the  MCO/PHP.  Fee- 
for-service  (FFS)  providers  have  the  greatest  incentive  to  submit  accurate  and 
complete  data,  since  their  payment  depends  on  it. 

2.  Contract  requirements:  Data  are  more  likely  to  be  complete  when  they  are 
required  for  payment. 

After  reviewing  each  MCO/PHP 's  IS  Capabilities  Assessment,  the  EQRO  staff  will  record  their 
analytic  findings  on  a  standard  form  such  as  the  Information  Systems  Capabilities  Assessment 
for  Managed  Care  Organizations  and  Prepaid  Health  Plans  -  Reviewer  Worksheet  and  Interview 
Guide  (Reviewer  Worksheet  and  Interview  Guide)  found  in  Appendix  A.  A  form  such  as  the 
Reviewer  Worksheet  and  Interview  Guide  serves  to  document  the  findings  of  the  EQRO  staff 
when  reviewing  the  IS  Capabilities  Assessment  for  each  MCO,  and  to  identify  those  issues  to  be 
addressed  in  Step  2,  the  follow-up  interview  with  MCO/PHP  personnel. 

Step  2:  Interview  personnel  at  each  MCO/PHP  to  augment  information  obtained  through 
the  standardized  IS  Capabilities  Assessment. 

Whether  the  EQRO  is  reviewing  a  previous  assessment  of  MCO/  PHP  IS  capabilities,  or  whether 
it  has  asked  the  MCO/PHP  to  complete  a  new  Information  Systems  Capabilities  Assessment  such 
as  that  in  Appendix  A,  the  written  descriptions  of  IS  capabilities  submitted  by  the  MCO/PHP 
must  be  reviewed  and  supplemented  by  conversations  with  the  MCO/PHP  staff  to  clarify  or 
gather  more  detailed  information  regarding  the  MCO's  /  PHP's  IS  capabilities.  The  EQRO  staff 
will  interview  appropriate  MCO/PHP  staff  using  a  standard  interview  protocol  such  as  the 
MCO/PHP  IS  Capabilities  Assessment  -  Reviewer  Worksheet  and  Interview  Guide  found  in 
Appendix  A.  However,  all  information  submitted  by  the  MCO/PHP  on  its  Information  Systems 
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Capabilities  Assessment  might  not  need  further  clarification.  In  addition,  not  all  questions  in  the 
Interview  Guide  may  need  to  be  discussed  with  MCO/PHP  staff  with  respect  to  encounter  data 
validation.  This  is  because  assessment  of  information  systems  can  be  conducted  for.  different 
purposes;  e.g.,  validation  of  performance  measures  and  determining  compliance  with  MCO/PHP 
structure  and  operational  standards.  Further,  some  questions  in  the  Reviewer  Worksheet  and 
Interview  Guide  will  need  to  be  reworded  for  newly  formed  MCOs/PHPs.  However,  the 
following  areas,  at  a  minimum,  should  be  fully  described  either  in  the  MCO/PHP' s  written 
documentation  of  its  IS  or  through  subsequent  follow-up  discussions  between  the  MCO/PHP  and 
the  EQRO. 

Information  Systems:  Data  Processing  and  Procedures 

1 .  Data  Base  Management  System  (DBMS)  Type 

2.  Programming  language 

3.  Updating  the  program  to  meet  changes  in  State  requirements 
Claims/Encounter  Processing 

1 .  Overview  of  the  processing  of  encounter  data  submissions 

2.  Completeness  of  the  data  submitted 

3.  Policies/procedures  for  audits  and  edits 

Claims/Encounter  System  Demonstration 

1 .  Processes  for  merges  and/or  transfer  of  data 

2.  Processes  for  encounter  data  handling,  logging  and  processes  for  adjudication 

3.  Audits  performed  to  assure  the  quality  and  accuracy  of  the  information  and  the 
timeliness  of  processing 

4.  Maintenance  and  updating  of  provider  data 

Enrollment  Data 

1 .  Verification  of  claims/encounter  data 

2.  Frequency  of  information  updates. 

3.  Management  of  enrollment/disenrollment  information 

Based  on  this  review  of  the  MCO's  /  PHP's  IS  capabilities,  the  EQRO  should  note  for  each 
encounter  type  listed  in  the  Acceptable  Error  Rates  Specification  Form  (described  previously) 
whether  or  not  there  are  concerns  about  certain  types  of  encounter  data,  and  note  it  in  the  fourth 
column  in  the  chart  below.  This  should  trigger  further  investigation. 
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Acceptable  Error  Rates  Specif  cations  and  Identified  Areas  of  Concern  Form 

Encounter  Type 

Error  Type 

Acceptable  Error  Rate 

Area  of  Concern 
(Yes  /  JNo) 

Office  Visit  -  (excludes  dental 
and  mental  health  /  substance 
abuse  visits) 

Missing 

^  0/ 
<  70 

Surplus 

0/ 

<  /o 

Erroneous 

0/ 

<  /o 

Office  Visit  -  mental  health  / 
substance  abuse 

Missing 

/o 

ourpius 

<-  0/ 

<~  /o 

raroneous 

^  /o 

Office  Visit  -  dental 

ivllSSUlg 

^  /o 

Surplus 

<  /o 

Erroneous 

^  0/ 

<  70 

Inpatient  admission  -  (excludes 
mental  health  /  substance  abuse 
visits) 

Missing 

^  0/ 

<  /o 

Surplus 

0/ 

<  /o 

Erroneous 

<r  o/ 
/o 

Inpatient  admission  -  mental 
health  /  substance  abuse 

IvllSSUlg 

/o 

Surplus 

«r-  0/ 

^  /o 

Erroneous 

<r-  0/ 
<  70 

Other  types  of  encounters  as 
specified  by  the  State;  e.g., 
laboratory,  pharmacy,  physical 
therapy. 

Missinff 

<  % 

Surplus 

<  % 

Erroneous 

<  % 

The  EQRO  should  add  as  many 
additional  rows  to  this  chart  as 
needed  to  incorporate  all  types 
of  encounters  specified  by  the 
State. 

Missing 

<  % 

Surplus 

<  % 

Erroneous 

<  % 

ACTIVITY  3:         Analyze  electronic  encounter  data  for  completeness  and  accuracy. 

This  activity  represents  the  core  of  the  process  that  the  EQRO  will  use  to  test  the  validity 
(completeness  and  accuracy)  of  the  encounter  data.  Once  the  steps  in  this  activity  have  been 
completed,  the  EQRO  and  the  State  will  have  an  excellent  assessment  of  whether  the  data  can  be 
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used  for  analysis.  If  the  EQRO  is  unsure  of  the  quality  of  the  encounter  data  at  the  completion  of 
Activity  3,  then  it  should  not  proceed  to  the  medical  record  review  activity  (Activity  4).  Rather,  it 
should  either  review  the  steps  of  this  activity  and  identify  areas  where  information  did  not  satisfy 
the  EQRO  or  it  should  seek  additional  assistance  to  determine  why  there  is  uncertainty  about  the 
quality  of  the  encounter  data.  If  the  steps  in  Activity  3  are  completed  thoroughly  and  accurately, 
there  should  be  little  doubt  about  the  quality  and  usefulness  of  the  submitted  encounter  data. 

In  this  activity,  the  EQRO  undertakes  analysis  of  each  MCO's  and  PHP's  encounter  data  through 
four  steps.  Information  obtained  from  these  four  steps  and  the  previously  conducted  IS 
Capabilities  Assessment  and  the  Structured  Interview,  should  yield  four  classes  of  information 
available  for  each  MCO/PHP: 

1.  General  magnitude  of  missing  encounter  data.  Evidence  of  whether  the  MCO/PHP 
has  been  unable  to  submit  any  encounter  data  and  reasons  for  failures,  such  as  the  inability 
to  process  the  encounter  data  without  edits. 

2.  Types  of  potentially  missing  encounter  data.  MCOs/PHPs  which  have  sub-capitated  or 
sub-contractor  relationships  with  providers  often  experience  difficulty  in  receiving 
information  from  those  providers.  Knowledge  of  the  MCO's/PHP's  contractual 
relationships  with  providers  will  help  identify  specific  areas  to  investigate  for  missing 
services. 

3.  Overall  data  quality  issues.  Identification  of  data  quality  problems  such  as  inability  to 
process  or  retain  certain  fields  on  the  encounter  data  record.  Some  MCOs/PHPs  may  not 
currently  have  room  in  their  systems  to  maintain  all  the  information  which  is  expected  to 
be  submitted. 

4.  MCO/PHP  data  issues.  Problems  with  how  the  files  are  compiled  and  submitted  to  the 
States. 

Step  1:  Analyze  information  from  the  IS  Capabilities  Assessment  and  the  follow- 

up  structured  interview  with  MCO/PHP  staff. 

At  the  completion  of  Activity  2,  the  EQRO  will  have  an  excellent  description  of  the  MCO/PHP's 
IS  and  should  know  what  to  expect  when  the  MCO/PHP's  data  files  are  investigated.  The 
information  from  Activity  2  is  incorporated  into  a  plan  for  testing  the  quality  of  the  data.  This  plan 
specifies  the  areas  that  will  be  tested  in  the  data  (the  areas  of  investigation)  and  the  expected 
results.  Having  such  a  plan  ensures  that  parts  of  the  data  quality  review  are  not  overlooked.  For 
example,  it  is  expected  that  MCOs  that  pay  a  substantial  portion  of  their  primary  care  providers  on 
a  capitated  basis  will  have  a  lower  encounter  data  submission  rate  than  MCOs  that  pay  their 
primary  care  providers  predominantly  on  a  FFS  basis.  As  a  part  of  a  data  quality  test  plan, 
provider  groups  would  need  to  be  identified  by  their  type  of  payment  and  rates  of  outpatient  visits 
per  eligible  beneficiary  calculated.  The  rates  of  outpatient  visits  would  then  be  compared  to  test 


HCFA/CMSO/QPMG/DQSM:  August  13,  2000  DRAFT 


Page  1 1 


the  assumption  that  capitated  providers  have  a  lower  rate  of  encounter  submission  than  FFS 
providers.  If  this  assumption  is  not  supported  by  the  data,  and  the  outpatient  visit  rates  differ  from 
the  benchmark,  then  other  questions  need  to  be  answered  about  the  MCO's  processing  of  FFS 
claims.  This  is  an  example  of  how  the  information  from  Activity  2  is  incorporated  into  Activity  3, 
Step  1 .  Other  questions  and  issues  can  be  addressed  in  the  same  way. 

Using  information  provided  by  the  State  in  Activity  1 ,  and  information  obtained  from  the 
MCO/PHP  through  the  IS  Capabilities  Assessment  and  the  Structured  Interview,  the  EQRO  staff 
will  develop  a  data  quality  test  plan  that  will: 

■  adjust  their  own  error  detection  programming  specifications  to  reflect  those  used 
by  the  MCO/PHP; 

■  adjust  their  own  report  specifications  to  match  those  used  by  the  MCO/PHP; 

■  create  ad  hoc  data  investigation  specifications,  based  on  the  information  gleaned 
from  the  IS  Capabilities  Assessment  and  the  interview; 

■  create  ad  hoc  report  writing  specifications;  and 

■  compile  notes  to  assist  in  data  interpretation. 

Step  2:  Inspect  the  MCO's/PHP's  Encounter  Data  files 

Step  2  and  Step  3  of  this  Activity  are  closely  inter-twined.  To  make  the  steps  clearer,  they  have 
been  broken  into  two  parts  because  having  two  steps  more  accurately  reflects  the  way  a  standard 
data  quality  review  process  would  occur.  When  data  are  reviewed  for  accuracy  and  completeness, 
they  are  subjected  to  a  macro  and  micro  analysis.  These  steps  are  described  separately  to  prevent 
the  EQRO  from  rushing  forward  to  generate  a  large  number  of  reports  and  analyses  before  the 
basic  integrity  of  the  data  have  been  verified.  Step  2  represents  the  macro  analysis;  Step  3 
represents  the  micro. 

Step  2  describes  a  basic  integrity  check  of  the  data  files.  It  answers  the  questions:  Are  there  data? 
Do  they  generally  fit  with  expectations?  Are  they  of  sufficient  basic  quality  to  proceed  with  more 
complex  analyses?  In  general,  all  of  the  analysis  that  is  required  in  Step  2  should  be  highly 
automated  and  generated  as  a  standard  data  review  process.  The  analysis  required  in  Step  2  can  be 
separately  performed  on  each  of  the  different  encounter  data  files  (e.g.,  hospital,  dental, 
ambulatory,  etc.)  for  each  of  the  data  fields  in  those  files,  while  the  analysis  described  in  Step  3 
requires  that  encounter,  eligibility,  and  provider  data  be  linked  together.  The  EQRO  will  obtain 
the  encounter  data  to  analyze  either  by  accessing  the  State's  information  system  or  by  receiving 
from  the  State  an  encounter  data  extract  from  the  States'  data  system  that  replicates  the  data  that 
the  State  has.  In  the  latter  case,  the  EQRO  will  access  the  data  using  its  own  analytical  processes. 
The  EQRO  will  inspect  the  files  and  perform  the  following  activities: 

1 .        Assure  that  the  enrollment  information  that  the  State  transfers  to  the  MCO  is 

accurately  incorporated  into  the  MCO  information  system  and  is  being  reported 
back  to  the  State  correctly.  In  many  cases,  MCO  information  systems  do  not  use 
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the  State's  Medicaid  identifier  as  the  means  for  tracking  enrollees.  When  the 
encounter  data  are  reported  to  the  State,  the  Medicaid  ID  must  be  reattached  to  the 
data  file.  In  this  step,  the  EQRO  will  verify  that  the  Medicaid  IDs  are  being 
reported  correctly.  As  an  additional,  but  not  required,  step,  the  EQRO  could 
compare  the  encounter  data  file  to  a  State  eligibility  file  and  check  for  accuracy  of 
the  IDs,  and  other  eligibility  information;  (e.g.,  age,  sex  and  eligibility  category). 
This  step  is  optional  since  in  the  majority  of  situations,  the  encounter  data  file  that 
the  EQRO  is  reviewing  has  been  edited  in  the  State  system  -  where  eligibility 
checking  is  one  of  the  core  elements.  This  step  should  be  considered  in  those 
situations  where  the  encounter  data  have  not  been  edited  in  the  State  system. 4  In 
addition,  the  EQRO  will  determine  whether  there  are  encounter  data  for  the 
majority  of  beneficiaries,  rather  than  a  large  volume  of  data  but  only  a  few  IDs. 
The  primary  focus  of  this  step  is  the  verification  of  the  correct  eligibility  numbers. 
In  other  parts  of  this  activity,  analysis  will  be  done  to  ensure  that  the  scope  and 
volume  of  services  are  consistent  with  the  eligibles. 

2.  Apply  general  edit  and  consistency  checks,  such  as  verifying  that  critical  fields 
contain  non-missing  values  and  that  values  are  consistent  across  fields;  e.g., 
pregnancy  and  related  diagnoses  and  procedures  are  for  individuals  whose  sex  is 
coded  as  female. 

3.  Inspect  the  data  fields  for  general  validity  (i.e.,  information  for  each  critical  field  is 
within  required  ranges,  and  the  volume  of  data  is  consistent  with  the  MCO/PHP's 
enrollment). 

4.  Capture  more  detailed  validity  information  from  the  encounter  data  fields  used  for 
reporting  purposes. 

The  analysis  includes  a  review  of  each  data  element  and  a  general  review  of  the  volume  of  data  by 
type  or  place  of  service.  This  review  concentrates  on  two  areas:  field  validation  and  completeness: 

1 .  Field  validation 

•  Percent  present:  required  data  fields  are  present  on  the  file  and  have  information  in 
that  field 

•  Percent  valid:  data  in  the  field  are  of  the  requested  type;  i.e.,  numeric  fields  have 
numbers,  character  fields  have  characters,  etc. 

•  Percent  valid  values:  In  those  fields,  the  values  are  the  expected  values;  e.g.,  are 
there  valid  ICD-9  codes  in  the  diagnosis  field,  not  just  random  numbers?  This 


In  a  few  States,  Medicaid  programs  have  chosen  to  by-pass  the  MMIS  and  have  MCOs  submit  data 
directly  to  an  outside  source.  In  these  cases  the  eligibility  checking  may  not  be  as  intense  as  that  done  within  the 
MMIS.  In  those  cases  the  EQRO  might  choose  to  add  this  additional  eligibility  review. 
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review  requires  comparing  the  specific  field  to  sources  of  information  showing 
valid  values. 5 

Field  validation  will  require  the  use  of  State  standards.  Examples  of  eventual  State 
standards  are  found  in  Attachment  1 . 

2.  Data  completeness 

•  Distribution  by  service  type:  determine  whether  there  are  data  distributed  as 
expected  in  the  large  data  types:  institutional,  provider,  pharmacy,  dental,  etc. 

•  Across  time:  determine  the  data  volumes  by  month.  Are  all  data  types  seen  in  all 
months?  Is  the  data  volume  consistent  across  the  months? 

Step  3:         Generate  and  Review  Analytic  Reports 

Using  simple  statistical  procedures  such  as  measures  of  central  tendency,  univariate  descriptive 
statistics  and  bivariate  distributions,  the  EQRO  will  analyze  the  data  to  obtain  a  "data  validity" 
overview  of  each  MCO's/PHP's  encounter  data.  This  process  will  analyze  and  interpret  data  on: 
1)  submitted  fields,  2)  volume/consistency  of  encounter  data,  and  3)  utilization  rates. 

Analyzing  and  Interpreting  Data  in  Submitted  Fields 

There  are  three  questions  to  be  addressed  in  a  field-specific  review: 

1 .  Is  there  information  in  the  field,  and  is  that  information  of  the  type 
requested?  Each  field  will  have  a  definition  which  will  include  data  type  (alpha, 
numeric,  mixed)  and  size.  The  fields  must  be  checked  to  determine  whether  the 
information  is  of  the  correct  type  and  size.  For  example,  if  ICD-9  diagnosis  codes 
have  been  requested,  the  field  should  have  5  digits.  If  CPT-4  codes  are  requested, 
the  field  should  have  4  digits.  If  the  State's  Medicaid  beneficiary  ID  is  requested, 
the  field  should  contain  the  correct  number  of  letters  and  digits. 

2.  Are  the  values  valid?  When  compared  to  an  external  standard,  are  the  values  in 
the  field  valid  for  that  standard?  For  instance,  if  ICD-9  diagnosis  codes  have  been 
requested,  are  the  values  in  the  diagnosis  field  valid  ICD-9  diagnosis  codes?  A 
field  could  have  5  digits  in  it  but  those  digits  might  not  represent  valid  codes. 

Findings  for  questions  1  and  2  could  be  recorded  on  a  standard  form  such  as  that  below: 


5  This  is  a  place  where  the  information  from  the  IS  Assessment  becomes  critical.  One  of  the  things  that  will 
be  asked  is  which  version  of  the  ICD-9  codes  is  used  by  the  MCO.  Often  the  State  and  the  MCO  maintain  different 
versions  of  these  files  resulting  in  values  being  considered  invalid  when  they  really  are  valid. 
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Sample  Form  for  Recording  Evaluation  of  Submitted  Fields 

Required  Field 

Information 
present 

Correct  type  of 
information 

Correct  size  of 
information 

Presence  of 
valid  value  ? 

# 

% 

# 

% 

# 

% 

# 

% 

Enrollee  ID 

Plan  ID 

Provider  ID 

Principal  Diagnosis 

Procedure  Code 

Date  of  Service 

Units  of  Service 

Others  (continue 
adding  fields  as 

3.       Are  the  values  reasonable?  A  frequency  distribution  of  the  values  needs  to  be 
developed  and  then  compared  to  an  external  standard  to  determine  whether  the 
values  make  sense  for  the  submitted  population.  For  instance,  if  one  of  the 
required  fields  is  "place-of-service,"  there  should  be  a  reasonable  distribution 
between  inpatient  hospital,  outpatient  hospital,  emergency  room  and  physician 
office.  In  this  data  review,  the  values  in  the  fields  could  have  passed  the  test  in 
steps  1  and  2,  but  failed  at  this  stage.  A  plan  could  submit  data  with  a  single  valid 
value  in  a  field;  these  data  would  pass  steps  1  and  2  but  fail  step  3. 

Analyzing  and  Interpreting  Volume  /  Consistency  of  Encounter  Data 

This  type  of  evaluation  provides  basic  statistics  on  the  encounter  data.  It  describes,  among  other 
things,  the  number  of  Medicaid  enrollees,  the  number  of  encounters,  and  counts  and  totals  for 
various  demographic  subgroups,  diagnoses  and  types  of  services.  The  EQRO  should  run 
frequency  distributions  on  specific  fields  as  well  as  on  the  variables  created  explicitly  for  data 
validation  reporting  purposes.  The  EQRO  may  also  run  distributions  on  subsets  of  variables  and 
observations  where  the  result  indicates  potential  data  validity  concerns.  For  instance,  a  subset  of 
rates  of  outpatient  services  by  provider  zip  code  might  highlight  missing  zip  codes  in  the  edit  file 
which  results  in  the  rejection  of  all  encounters  with  that  zip  code.  This  initially  looks  like  an 
overall  low  rate  of  services  but  by  looking  at  a  subset  of  a  specific  field  and  checking  for 
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reasonableness,  a  different  problem  is  detected.  The  EQRO  also  should  generate  univariate 
statistics  (e.g.,  means,  medians  and  modes)  as  appropriate  on  continuous  and  discrete  data  fields. 
The  output  produced  for  these  reports  should  be  checked  for  both  reasonableness  and  to  detect 
specific  problems,  such  as  entire  categories  of  data  missing  from  the  regular  data  submissions. 

The  EQRO  should  also  analyze  encounter  data  for  other  volume/consistency  dimensions.  These 
dimensions  can  include  time,  provider  type,  type  of  service,  and  demographic  groupings,  but 
States  may  have  additional  dimensions  (aid  category)  which  also  need  to  be  included.  This 
information  allows  the  EQRO  to  look  for  trends  such  as  the  following: 

■  Time  -  This  analysis  would  examine  encounter  data  both  by  service  date  and  by 
processing  date  to  check  consistency.  MCOs/PHPs  often  have  problems  processing 
encounter  data  and  in  many  cases  these  claims  are  processed  sporadically.  When  such  a 
situation  is  present,  it  can  often  be  an  indication  of  other  problems  within  the  MCO/  PHP's 
information  system.  After  establishing  the  length  of  time  between  service  dates  and 
processing  dates  by  MCO/PHP,  the  EQRO  could  compare  these  with  existing  benchmarks 
for  data  submission  and  processing. 

■  Provider  -  Encounter  data  validation  can  verify  the  presence  of  encounter  data  for  all 
provider  types  and  determine  if  there  are  significant  fluctuations  in  patient  visits  per  time 
period.  In  addition,  information  collected  during  the  capabilities  assessment  will  be  used 
to  identify  missing  encounter  data  for  specific  provider  types.  The  distribution  of 
encounter  data  will  be  compared  by  provider  type  with  the  benchmark  information 
described  above. 

■  Service  Type  -  The  EQRO  should  verify  whether  ancillary  services  (e.g.,  labs,  x-rays, 
therapy,  etc.)  are  evenly  represented  as  visits.  The  clinical  connection  between  the  use  of 
services  is  being  evaluated.  If  a  Medicaid  beneficiary  is  receiving  x-ray  or  lab  tests,  one 
would  expect  to  see  an  office  visit  (or  perhaps  a  hospital  admission)  in  the  same  time 
frame.  Other  areas  where  reasonableness  between  the  encounter  data  should  be  tested 
include:  1 )  relationship  of  outpatient  visits  to  number  of  prescriptions,  2)  relationship  of 
primary  to  specialty  care  visits,  and  3)  outpatient  services  associated  with  inpatient 
admissions.  Examination  of  the  data  in  this  way  will  reveal  whether  there  are  missing 
encounters. 

■  Age-  and  Sex- Appropriate  Diagnoses  and  Services  -  ine  EQRO  will  determine  if  the 
diagnoses  and  services  reflect  expected  care  by  age  and  sex.  As  an  example,  one  would 
expect  sex-specific  diagnoses  (such  as  endometriosis  or  undescended  testes)  and 
procedures  (such  as  deliveries  or  hysterectomies)  would  have  the  patient's  sex  coded 
correctly.  Conversely  one  would  expect  that  men  and  women  would  not  have  procedures 
coded  that  cannot  be  performed  on  a  person  of  that  sex. 

As  part  of  the  review,  the  EQRO  will  find  it  helpful  to  display  the  data  quality  findings 
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graphically.  It  is  nearly  always  true  that  in  these  situations,  "a  picture  is  worth  a  thousand  words." 
These  graphs  will  be  useful  internally  for  identifying  issues,  and  externally  for  conveying  the 
results  of  the  data  quality  review.  An  example  of  the  type  of  chart  that  might  be  generated  is  a 
frequency  distribution  of  the  number  of  encounters  per  Medicaid  enrollee.  This  chart  shows  how 
many  enrollees  had  zero,  one,  two,  three,  etc.  encounters.  This  chart  can  also  be  replicated  by 
different  age,  sex,  and  race  groupings,  to  see  whether  there  are  differences  along  these 
dimensions.  These  charts  should  be  generated  for  all  MCOs/PHPs  in  the  aggregate,  and  for  each 
MCO  separately  so  that  issues  with  a  specific  plan  can  be  identified. 

Analyzing  and  Interpreting  Utilization  Data 

The  EQRO  also  should  routinely  compile  and  review,  on  a  periodic  (e.g.,  monthly,  bi-monthly, 
quarterly  or  other  periodicity  as  directed  by  the  State )  basis,  statistics  displaying  information  on 
utilization  rates  overall  and  by  specific  diagnosis,  procedure,  service  and  provider  types  when 
appropriate.  These  reports  initially  should  be  generated  both  for  each  MCO/PHP  and  on  the  entire 
encounter  data  set  for  all  MCOs/PHPs  together  to  account  for  problems  associated  with  small 
numbers  of  encounters  for  certain  MCOs/PHPs.  During  the  program  start-up  phase,  many 
MCOs/PHPs  may  have  very  few  encounters  for  some  diagnoses  and  services,  which  would  make 
their  rates  statistically  imprecise. 

One  method  for  estimating  the  completeness  of  the  encounter  data  for  each  MCO/PHP  is  to 
benchmark  the  utilization  rates  for  a  given  MCO/PHP  against  utilization  statistics  from  other 
sources.  These  benchmarks  would  be  incorporated  into  the  MCO/PHP-specific  utilization  reports. 
Utilization  rates  could  be  broken-out  by  patient  demographics,  diagnosis,  type  of  service  and  type 
of  provider. 

Step  4:  Compare  findings  to  State-identified  standards 

The  EQRO  will  next  compare  the  encounter  data  submitted  by  each  MCO/PHP  to  standards  and 
benchmarks  that  are  identified  by  the  State.  These  standards  can  be  obtained  from  a  number  of 
different  sources,  including:  aggregate  encounter  data  from  all  Medicaid  MCOs/PHPs  in  that  State 
or  other  States  which  are  considered  to  be  comparable,  historical  FFS  Medicaid  data  in  that  State, 
or  data  from  a  State's  Primary  Care  Case  Management  (PCCM)  program.  The  State  may  also  look 
to:  commercial  managed  care  plans,  national  standards,  or  other  benchmarks.  The  State  will  need 
to  identify  such  standards  and  document  them  using  a  form  such  as  that  found  in  Attachment  2: 
Table  o^  Benchmark  Utilization  Rates.  States  will  also  need  to  specify  acceptable  variation  from 
these  standards.  Both  the  standards  and  acceptable  variations  from  the  standards  can  be  made 
more  stringent  over  time. 

For  instance,  when  comparing  encounter  data  utilization  rates  to  a  Medicaid  FFS  benchmark,  one 
might  expect  to  see  a  drop  in  emergency  room  utilization  per  member  month  under  managed  care. 
However,  large,  downward  swings  in  other  types  of  utilization  (e.g.,  >30  percent  drops  in 
ambulatory  care)  may  indicate  incomplete  encounter  data  rather  than  a  change  in  provider  practice 


HCFA/CMSO/QPMG/DQSM:  August  13,  2000  DRAFT 


Page  17 


patterns.  The  EQRO  should  test  their  assumptions  about  changes  in  utilization  under  Medicaid 
managed  care  with  the  MCOs/PHPs  and  with  the  State,  as  a  further  test  of  the  completeness  of  the 
encounter  data. 

The  results  of  ACTIVITY  3  are  used  to  form  the  basis  for  a  long-term  monitoring  strategy  for 
assessing  the  quality  of  the  encounter  data.  As  the  data  improve  over  time  due  to  monitoring  and 
validation,  the  EQRO  will  be  able  to  design  targeted  validation  strategies  by  using  analytic  testing 
on  the  encounter  data  files  to  identify  problem  areas  requiring  medical  record  review  validation, 
thus  allowing  the  conservation  of  resources  by  avoiding  unfocused  "broadside"  medical  record 
review. 


ACTIVITY  4:  Review  of  medical  records  for  confirmation  of  findings  of  analysis  of 

encounter  data 

Medical  record  review  can  provide  additional  verification  of  the  information  obtained  from  the 
preceding  analysis  of  electronic  encounter  data  for  accuracy  and  completeness  (Activity  3). 
However,  medical  record  validation  is  a  complex  and  resource-intensive  process.  While  it  can 
easily  be  used  to  validate  specific  areas  of  concern,  it  is  not  an  efficient  method  for  performing  a 
more  general  validation  of  encounter  data.  As  stated  at  the  beginning  of  Activity  3,  if  the  EQRO 
is  unsure  of  the  quality  of  the  encounter  data  at  the  completion  of  Activity  3,  then  it  should  not 
proceed  to  medical  record  review.  Rather,  it  should  either  review  the  steps  in  Activity  3  and 
identify  areas  where  information  did  not  satisfy  the  EQRO  or  it  should  seek  additional  assistance 
to  determine  why  there  is  uncertainty  about  the  quality  of  the  encounter  data.  If  the  State  is  in  the 
initial  phase  of  collecting  encounter  data  from  its  MCO  or  PHPs,  there  may  be  a  time  lag  of  as 
much  as  three  years  before  the  encounter  data  are  of  sufficient  completeness  and  accuracy  to 
warrant  the  investment  in  medical  record  review 

Further,  medical  record  review  should  not  be  used  to  validate  information  that  is  collected  at 
another  source  and  considered  to  be  more  accurate.  For  example,  information  collected  as  part  of 
eligibility  determination  is  considered  the  primary  source  for  demographic  information  such  as 
patient  age,  sex,  and  race.  This  eligibility  data  should  be  used  as  the  source  of  demographic 
information  when  validating  encounter  files. 

In  this  protocol,  the  following  assumptions  are  made  with  respect  to  medical  record  review: 

•  Medical  record  review  for  encounter  data  validation  is  being  performed  independently  of 
medical  record  review  for  evaluation  of  performance  measures  or  other  purposes, 
however,  these  reviews  could  be  coordinated  in  an  effort  to  reduce  the  burden  on  the 
MCOs/PHPs. 

•  When  medical  record  review  should  begin,  and  how  frequently  it  should  be  performed, 
will  be  decided  by  the  State. 
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•         One  medical  record  review  is  determined  by  the  State  to  be  appropriate,  the  EQRO  will 
draw  a  sample  of  medical  records  for  validation  on  a  regular  and  periodic  basis  specified 
by  the  State. 

Validating  encounter  data  using  medical  records  must  be  approached  as  if  it  were  a  research 
question.  There  needs  to  be  clear  hypotheses,  well  defined  populations,  and  stated  error 
tolerances.  A  rigorous  research  design  ensures  that  the  results  of  this  resource  intensive  effort  will 
be  meaningful  and  useful. 

The  approach  to  medical  record  validation  depends  on  the  questions/hypotheses  to  be  addressed. 
Depending  on  the  stated  hypotheses,  one  would  begin  with  a  sample  of  encounters,  enrollees,  or 
both.  The  table  below  illustrates  how  two  different  questions  are  answered  by  using  different 
sampling  universes. 


Examples  of  questions  to  be 
addressed 


Is  the  information  found  on 
the  encounters  accurate  when 
compared  to  the  medical 
record? 


sampling  universe 


Encounters 


Enrollees 


Use  sample  of  encounters 


Are  there  electronic 
encounters  for  all  the  services 
that  were  provided  to 
enrollees? 


Use  sample  of  enrollees 


Another  possible  question  may  be:  "Are  the  encounters  fully  coded  for  all  diagnoses?"  Often 
when  providers  submit  encounters,  they  only  provide  the  primary  diagnosis  code,  since  that  is 
usually  sufficient  for  payment.  Another  question  might  be  "Are  the  procedures  fully  coded?" 
Providers  might  record  on  the  encounter  form  only  those  procedures  that  historically  have  affected 
reimbursement.  In  such  cases,  additional  procedures  performed  can  only  be  discovered  by 
reviewing  the  medical  record.  In  this  type  of  medical  record  validation,  a  sample  of  encounters  is 
selected,  and  the  medical  records  for  those  encounters  are  reviewed. 

When  one  wants  to  determine  whether  all  encounters  have  been  received  for  services  that  were 
delivered,  then  a  sample  of  enrollees  is  selected,  and  their  medical  records  are  reviewed  against 
encounters.  In  this  case,  the  EQRO  would  sample  MCO/PHP  enrollees  rather  than  encounters  for 
a  specific  time  period.  Services  recorded  in  the  medical  record  would  be  matched  with  those 
found  on  the  encounters  to  assess  the  completeness  and  accuracy  of  encounter  data.  To  reduce  the 
number  of  medical  records  that  must  be  located  for  each  enrollee,  the  EQRO  should  consider 
limiting  medical  record  review  to  a  specific  type  of  encounter,  such  as  inpatient  admissions  or 
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physician  office  visits. 

Where  medical  record  review  is  performed  is  a  complicated  question  and  each  solution  has  its 
own  strengths  and  weaknesses.  In  all  cases,  the  medical  record  must  be  located,  the  reviewers 
must  be  trained  and  experienced,  the  confidentiality  of  patient  records  must  be  maintained,  and 
costs  must  be  minimized.  Balancing  these  four  requirements  is  difficult  and  results  in  varied 
solutions  that  fit  with  the  specifics  of  each  particular  situation.  In  many  cases,  reviewers  decide  to 
look  at  medical  records  away  from  the  site  of  health  care  delivery  and  request  that  providers  copy 
charts  and  send  them  to  a  central  location.  This  approach  places  a  large  amount  of  the 
responsibility  for  locating  the  charts  on  the  plans  and  providers,  but  also  increases  the  number  of 
charts  that  are  located.  Providers  should  be  compensated  for  the  direct  costs  of  copying  and 
submitting  the  files. 

Step  1 :  Sampling  for  medical  record  review 

When  medical  record  review  is  to  be  undertaken,  the  EQRO  may  initially  choose  to  conduct 
"trial"  reviews  of  the  encounter  data  using  medical  records  previously  obtained  for  other  purposes, 
such  as  focused  studies  of  clinical  care  topics.  This  trial  review  would  provide  an  opportunity  for 
the  EQRO  to  acquire  experience  in  using  medical  records  specific  to  individual  MCO/PHPs  for 
encounter  data  validation,  as  well  as  highlighting  community  practice  patterns  with  regard  to 
developing  encounters  from  medical  records,  without  adding  unnecessary  administrative  burden 
to  the  MCO/PHPs. 

Once  medical  record  review  is  to  be  conducted  on  a  more  routine  basis,  the  size  of  the  sample 
must  be  determined  for  each  MCO/PHP.  The  size  of  the  sample  of  medical  records  that  must  be 
drawn  in  order  to  make  statistically  valid  inferences  regarding  the  validity  of  the  encounter  data 
depends  on  a  number  of  factors,  including: 

•  The  minimum  error  rate  the  State  wants  to  be  able  to  detect; 

•  The  frequency  with  which  the  State  wants  to  perform  the  review;  and 

•  The  subsets  of  encounter  data  the  State  intends  to  validate. 

Because  of  these  and  other  factors;  e.g.  the  size  of  an  MCO/PHP's  enrollment  and  previous 
validation  results,  it  may  be  statistically  appropriate  to  determine  different  sample  sizes  for  each 
MCO/PHP.  However,  it  maybe  operationally  more  efficient  (and  also  statistically  valid)  to 
specify  the  same  sample  size  for  all  MCO/PHPs.  The  EQRO  will  need  to  determine  the  sample 
size  for  each  MCO/PHP  either  as  directed  by  the  State,  or,  if  the  State  allows  the  sample  size  to  be 
determined  by  the  EQRO,  in  consultation  with  a  qualified  statistician.  Whether  one  sample  size  or 
multiple  sample  sizes  will  be  used  will  be  determined  by  the  State,  in  consultation  with  the 
EQRO. 

Once  the  sample  size  is  determined,  the  EQRO  will  select  this  number  of  enrollees  from  each 
MCO/PHP  for  medical  record  review  in  order  to  calculate  fault  rates  for  each  MCO/PHP.  The 
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sampling  must  be  performed  using  methodologically  sound  techniques  that  defend  against 
sampling  bias.  The  fault  rate  is  the  ratio  of  missing  and  erroneous  records  to  the  total  number  of 
encounters  that  took  place  during  the  time  period  being  examined.  A  fault  rate  can  be  calculated 
for  each  encounter  type. 

It  is  anticipated  that  fault  rates  initially  will  be  at  least  30  percent.  However,  each  MCO's/PHP's 
targeted  fault  rate  should  be  below  5  percent  (f<0.05)  for  each  time  period  examined,  and  if 
possible,  demonstrate  a  decrease  over  time.  Consequently,  for  each  time  period  examined,  the 
EQRO  will  test  the  hypothesis  that  there  is  no  difference  between  the  services  recorded  in  the 
medical  record  and  those  found  in  the  encounter  data  (i.e.,  the  "null"  hypothesis).  The  ability  of 
this  test  to  reject  the  null  hypothesis  (H0)  when  it  should  be  rejected  will  depend  on  the  size  of  the 
encounter  sample,  the  size  of  the  total  population  of  true  encounters,  and  the  MCO's/PHP's  actual 
fault  rate.  The  EQRO  should  set  sample  sizes  sufficient  to  estimate  the  fault  rate  for  each  type  of 
encounter  within  each  MCO/PHP,  with  equal  precision  for  each  time  period  to  be  studied. 

Step  2:  Review  medical  records  and  record  findings  on  a  standardized  worksheet. 

Fields  to  be  validated  through  medical  record  review  should  include  a  few  socio-demographic 
fields  (e.g.  date  of  birth,  sex)  that  may  be  needed  to  identify  the  correct  beneficiary.  Typically, 
when  medical  records  are  selected  for  review,  the  provider  is  given  the  patient's  name,  age,  and 
sex,  the  provider's  name,  and  the  target  dates  of  service.  This  information  helps  the  provider  find 
the  correct  medical  record.  When  the  comparison  is  done  between  the  encounter  data  and  the 
medical  record,  this  information  is  included  in  the  data  elements  that  are  reviewed  to  further 
ensure  that  the  correct  record  was  actually  reviewed.  The  demographic  information  on  the  medical 
record  is  not  considered  the  definitive  source  of  demographic  information.  It  is  included  here  to 
support  the  medical  record  review  process.  Other  fields  on  the  encounter  data  should  be  assessed 
so  that  it  is  possible  to  create  measures  of  access  and  quality  of  care.  These  data  elements  include: 
date  of  service  and  the  clinical  codes  (such  as  ICD-9-CM,  HCPCS  and  CPT  codes)  that  define 
diagnoses,  procedures,  and  other  services. 

For  diagnoses,  the  medical  record  review  staff  should  review  codes  based  on  the  diagnoses  stated 
by  the  provider  in  the  medical  record,  (not  based  on  diagnoses  indicated  by  their  own  judgement). 
Because  of  this,  medical  record  review  staff  should  be  experienced  clinical  coding  validators. 
Clinical  coding  validators  should  have  substantial  clinical  background,  including  anatomy  and 
physiology,  pathology,  microbiology,  pharmacology,  and  disease  process.  They  use  this  clinical 
understanding,  combined  with  their  knowledge  of  appropriate  coding  guidelines,  to  assign  the 
right  codes  to  a  record.  When  a  medical  record  lacks  sufficient  documentation  to  select  the  most 
specific  code(s),  clinical  coding  validators  may  consult  with  each  other  and  other  health  care 
professionals  to  answer  clinical  questions. 

Sources  of  coding  errors,  described  by  the  Institute  of  Medicine  in  a  study  of  the  reliability  of 
coded  data  in  the  National  Hospital  Discharge  Survey  included: 
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1 )  Incorrect  selection  or  sequencing  of  principal  diagnosis  codes, 

2)  Incorrect  selection  of  other  individual  codes, 

3)  Coding  diagnoses  or  procedures  not  documented  in  the  medical  record, 

4)  Errors  caused  by  mistakes  in  entering  the  data  into  a  database,  and 

5)  Failure  to  review  the  entire  medical  record. 

Building  on  these  ideas,  the  EQRO  should  develop  an  error  categorization  scheme  to  identify 
areas  of  incompleteness  and  inaccuracy.  Codes  assigned  by  the  clinical  coding  validators  will 
have  three  components,  as  separate  data  elements.  Data  can  be  reported  for  any  single  error 
component,  or  by  any  combination  of  these  codes.  These  error  components  are: 

Level:  Identifies  whether  an  encounter  is  present  in  the  database.  The  presence  or  absence 
of  an  encounter  determines  the  strategy  followed  by  the  reviewer  to  complete  the  review. 

Type:  Describes  if  codes  or  other  data  are  correctly  or  incorrectly  present  or  absent. 

Source:  Assigns  the  most  likely  reason  for  the  type  of  error  found.  While  this  may  be  a 
subjective  determination,  it  will  be  helpful  when  giving  feedback  to  MCOs  /  PHPs  so  that 
they  can  target  their  data  quality  improvement  processes. 

Certain  errors  may  be  designated  as  "critical"  for  the  purposes  of  an  audit.  These  can  be  defined 
differently  at  varying  points  in  time,  because  issues  that  are  critical  in  encounter  data  validation 
change  over  time.  For  example,  in  the  early  rounds  of  validation,  a  State  may  wish  to  focus  on 
diagnosis  and  procedure  codes,  and  not  on  physician  specialty  or  place-of-service.  These  fields  are 
important,  but  are  of  little  value  if  the  MCOs  are  not  able  to  produce  accurate  clinical  coding. 
Once  accurate  diagnosis  and  procedure  coding  is  taking  place,  knowing  accurate  specialty  and 
place  of  service  greatly  increases  the  value  of  the  data.  Having  "tiers  of  errors"  (e.g.,  critical, 
serious,  moderate,  etc.)  allows  the  State  to  move  ahead  with  using  encounter  data  that  are  not 
totally  complete  and  accurate.  This  is  an  important  goal.  In  future  years,  another  tier  of  more 
refined  error  types  will  be  added  to  the  critical  error  types.  Assigning  the  label  of  "critical"  to  the 
errors  will  be  determined  by  the  State. 

The  findings  of  each  medical  record  review  should  be  documented  on  a  standardized  form,  such 
as  the  Medical  Record  Review  Findings  Tool  for  Encounter  Data  Validation  found  as  Attachment 
3.  It  is  also  important  to  provide  documentation  guidelines  to  the  staff  performing  the  medical 
record  reviews.  These  guidelines  should  describe  exactly  how  to  document  the  findings  of  the 
medical  record  review.  The  guidelines  also  should  be  closely  linked  to  the  reporting  requirements 
and  the  data  elements  chosen  for  validation,  so  they  should  be  written  after  the  error  classification 
scheme  is  set.  Written  policies  for  interpretation  of  the  documentation  guidelines  should  include 
the  following: 

•  Directions  for  reviewing  medical  records; 

•  Instructions  on  what  to  do  when  faced  with  conflicting  documentation; 
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•  Instructions  on  what  to  do  when  no  code  can  be  readily  assigned; 

•  Use  of  optional  codes; 

•  Definitions  of  what  constitutes  errors,  and  how  to  document  them; 

•  List  and  location  of  approved  reference  materials  (coding  manuals,  medical  textbooks, 
etc.); 

•  Whom  to  consult  for  additional  assistance. 
ACTIVITY  5:  Submission  of  findings 

After  the  performance  of  Activities  1-  4,  the  EQRO  will  create  data  tables  that  display  summary 
statistics  for  the  information  obtained  from  these  activities  for  each  MCO/PHP.  Summarizing  the 
information  in  tables  makes  it  easier  to  evaluate,  and  highlights  issues  with  respect  to  the  accuracy 
and  completeness  of  encounter  data.  A  narrative  will  accompany  these  tables  which  will  highlight 
MCO/PHP-specific  issues. 

In  addition,  a  State  at  its  discretion,  may  direct  its  EQRO  to  undertake  technical  assistance  to  its 
MCOs/PHPs  to  improve  the  accuracy  and  completeness  of  encounter  data. 


END  OF  PROTOCOL 
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EXAMPLES  OF  RECOMMENDED  DATA  QUALITY  STANDARDS  FOR  EVALUATION  OF 
SUBMITTED  ENCOUNTER  DATA  FIELDS  (Physician  and  Other  Provider) 

Data  Element 

Expectation 

Validity  Criteria 

Enrollee  ID 

Should  be  valid  ID  as  found  in  the  State's 
eligibility  file.  Can  use  State's  ID  unless 
State  also  accepts  SSN. 

100%  valid 

Enrollee  Name 

Should  be  captured  in  such  a  way  that 
makes  separating  pieces  of  name  easy. 
There  may  be  some  confidentiality  issues 
that  make  this  difficult  to  obtain.  If 
collectable,  expect  data  to  be  present  and  of 

arvnn  mi 5*1 1 

85%  present.  Lengths  should  vary  and 

,1                                    1                       111                          1       1                        J  1 

there  should  be  at  least  some  last  names 
>8  digits  and  some  first  names  <  8 
digits.  This  will  validate  that  fields  have 
not  been  truncated.  Also  verify  that  a 

V110V1  r*fvrr*fvnt!3  0'fk  navf*  c\i  l^tict  a  mirlHIf* 
lllt^ll  JJCi CCIlLa^C  Have  cH  ICdol  a  1111U.U1C 

initial. 

Enrollee  Date  of 
Birth 

Should  not  be  missing  and  should  be  a  valid 
date. 

<  2%  missing  or  invalid 

MCO/PHP  ID 

Critical  Data  Element 

100%  valid 

Provider  ID 

Should  be  an  enrolled  provider  listed  in 
provider  enrollment  file. 

95%  valid  M 

^( 

Attending  Provider 
ID 

Should  be  an  enrolled  provider  listed  in 
provider  enrollment  file  (also  accept  the 
MD  license  number  if  listed  in  provider 
enrollment  file). 

>  85%  match  with  provider  file  using 
either  provider  ID  or  MD  license 
number 

Provider  Location 

Minimal  requirement  is  county  code,  with 
zip  code  being  strongly  advised. 

>  95%  with  valid  county  code 

>  95%  with  valid  zip  code  (if  available) 

Place  of  Service 

Should  be  routinely  coded,  especially  for 
physicians 

>  95%  valid  for  physicians 

>  80%  valid  across  all  providers 

Specialty  Code 

Coded  mostly  on  physician  and  other 
practitioner,  optional  on  other  types  of 
providers 

Expect  >  80%  non-missing  and  valid  on 
physician  or  other  applicable  provider 
type  claims  (e.g.  other  practitioners) 
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1 

principal  Diagnosis 

9 

Well  coded  except  by  ancillary  type 
providers 

>  90%  non-missing  and  valid  codes 
(using  lCD-y-CM  lookup  tables)  tor 
practitioner  providers  (not  including 
transportation,  lab  and  other  ancillary 
providers) 

Other  Diagnosis 

This  is  not  expected  to  be  coded  on  all 
claims  even  with  applicable  provider  types, 
but  should  be  coded  with  a  fairly  high 
frequency. 

90%  valid  when  present 

Date  of  Service 

Dates  should  be  evenly  distributed  across 
time 

If  looking  at  a  full  year  of  data,  5-7%  of 
the  records  should  be  distributed  across 
each  month. 

Unit  of  Service 
(Quantity) 

The  number  should  be  routinely  coded 

98%  non-zero 

<  70%  should  be  one  if  CPT  code  in 
range  99200-99215,  99241-99291 

Procedure  Code 

J 

This  is  a  critical  data  element  and  should 
always  be  coded. 

99%  present  (not  zero,  blank,  8-  or  9- 
iillea).  IUUto  snoula  be  valia,  Mate- 
approved  codes.  There  should  be  a  wide 
range  of  procedures  with  the  same 
frequency  as  previously  encountered. 

Procedure  Code 
Modifier 

This  is  important  to  pick  up  to  separate  out 
surgical  procedures/anesthesia/asst. 
Surgeon.  It  is  not  applicable  for  all 
procedure  codes 

>  20%  non-missing.  Expect  a  variety  of 
modifiers  both  numeric  (CPT)  and 
Alpna  (ntrLo).  l  be  more  common 
codes  which  should  appear  with  at  least 
a  minimal  frequency  are:  47  (anesthesia) 
and  80  (asst.  surgeon). 

EPSDT  Indicator 
(All  States  might  not 
have  this.) 

If  this  field  is  used,  the  beneficiary  should 
be  <  2 1  years  of  age  and  the  provider  should 
be  certified  to  administer  EPSDT  screens. 

95%enrollees<21; 

85%  providers  certified  as  EPSDT 

Patient  Discharge 
Status  Code 
(Hospital) 

Should  be  valid  codes  for  inpatient  claims 
with  the  most  common  code  to  be 
"Discharged  to  Home."  For  outpatient 
claims  it  can  be  coded  as  "not  applicable  ." 

For  inpatient  claims,  expect  >90% 
"Discharged  to  Home."  Expect  1  -5%  in 
all  other  values  (except  "not  applicable" 
or  "unknown"). 

Revenue  Code 
(Hospital) 

If  facility  uses  UB92  claim  form,  should 
always  be  present. 

100%  valid. 

Source:  The  MEDSTAT  Group 
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TABLE  OF  BENCHMARK  UTILIZATION  RATES 
(for  services  incurred  betvwen  XX/200x  and  YY/200x) 

Measure 

Value  from 
MA  FFS  or 
PCCM 

Value  from 
Comparable 
State  or  States 

Other 
Comparison 
Value 

Inpatient  Discharges 

Inpatient  LOS 

Overall 

By  high  volume  DRGs 

By  eligibility  category/patient  cohort 

Ambulatory  Surgeries 

Total  #  surgeries 

By  high  volume  CPT  codes  or  by 
ambulatory  surgery  categories 

Total  #  surgeries/ 1,000  enrollees 

By  high  volume  CPT  codes  or  by 
ambulatory  surgery  categories 

Number  of  Providers 

Primary  care  physicians 
Specialists 

Other  (e.g.,  mental  health  providers) 

Number  of  Enrollees 

Total  # 

By  eligibility  category 
By  age/sex  categories 

Number  of  Users  (i.e.,  enrollees  who  used 

services) 

1  otal  # 

By  eligibility  category 
By  age/sex  categories 

Visits 

Total  # 

#/enrollee 

#/user 

by  visit  categories  (e.g.,  well  child,  well  adult, 
ob/gyn,  mental  health,  substance  abuse,  etc.) 

Other  Services  (e.g.,  prescription  drug) 

Total  # 

#/enrollee 

#/user 

by  service  category 
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Attachment  3 

Sample  Medical  Record  Review  Findings  Tool 
for  Encounter  Data  Validation 


Medical  Record  Number._ 

Completion  Date:  

EQRO  Reviewer:  

Coder  Reviewer:  

Attending  Physician  Name: 
End  Date:  


Required  Review:  (Check  one) 

[     ]  Office  Visit  -  (excludes  dental  and  mental  health  /  substance  abuse  \isits) 
[     ]  Office  Visit  -  mental  health  /  substance  abuse 
[     ]  Office  Visit  -dental 

[     ]  Inpatient  admission  -  (excludes  mental  health  /  substance  abuse  \isits) 
!     ]  Inpatient  admission  -  mental  health  /  substance  abuse 

]  Other  types  of  encounters  as  specified  bythe  State;  e.g,  laboratory,  pharmacy,  physical 


therapy.  Specify 


Diagnosis  Codes  and  Descriptions 

Diagnosis  Code 

Match 

No  Match 

Diagnosis  Description 

a. 

b. 

c. 

d. 

If  the  diagnoses  in  the  record  do  not  match  the  billed  information,  write  the  correct  diagnosis 

description(s)  on  the  lines  provided. 

Patient  ID  Number: 

Patient  Name:  

Provider  Name:  


Visit  Dates:     Begin  Date: 
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Procedure  Codes  and  Descriptions 

Procedure 
Code 

Match 

No  Match 

y 

Procedure  Description                f  \ 

a. 

b. 

c. 

d. 

e. 

f. 

If  the  procedures  in  the  record  do  not  match  the  billed  information,  write  the  correct  procedure 
description(s)  in  the  spaces  provided. 

Revenue  Center  Codes  and  Descriptions 

Revenue 

Center 

Codes 

Revenue  Center 
Descriptions 

Match 

No 
Match 

Correct  Revenue 
Center  Description 

Correct  (j 
Code 

a. 

b. 

c. 

d. 

e. 

If  revenue  centers  in  the  record  do  not  match  the  billed  information,  write  the  correct  revenue  center 
Description(s)  in  the  spaces  provided. 

NOTE:  The  EQRO  should  tailor  and  add  to  this  form  to  address  all  data 
fields  under  review. 

END  OF  PROTOCOL  ATTACHMENTS 
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APPENDIX  A 


INFORMATION  SYSTEMS  CAPABILITIES  ASSESSMENT  FOR 
MANAGED  CARE  ORGANIZATIONS  /  PREPAID  HEALTH  PLANS 


I.  PURPOSE  OF  THE  ASSESSMENT 

Knowledge  of  the  capabilities  of  a  Managed  Care  Organization  (MCO)'s  or  Prepaid  Health  Plan 
(PHP)'s  Information  Systems  (IS)  is  essential  to  effectively  and  efficiently: 

•  validate  MCO/PHP  encounter  data, 

•  calculate  or  validate  MCO/PHP  performance  measures,  and 

•  assess  an  MCO/PHP's  capacity  to  well  manage  the  health  care  of  its  enrollees. 

The  purpose  of  this  assessment  is  to  specify  the  desired  capabilities  of  MCO/PHP  ISs  and  to  pose 
standard  questions  to  be  used  to  assess  the  strength  of  an  MCO/PHP  with  respect  to  these 
capabilities.  This  will  assist  an  EQRO1  to  assess  the  extent  to  which  an  MCO/PHP  IS  is  capable 
of  producing  valid  encounter  data,  performance  measures,  and  other  data  necessary  to  support 
quality  assessment  and  improvement,  as  well  as  managing  the  care  delivered  to  its  enrollees. 

Some  States  assess  the  capabilities  of  MCO/PHP  information  systems  as  part  of  pre-contracting, 
contract  compliance  or  contract  renewal  activities.  If  an  assessment  has  been  completed  by  the 
State  through  these  or  other  means;  e.g.,  private  sector  accreditation  or  performance  measures 
validation,  and  the  information  gathered  is  the  same  as  or  consistent  with  what  is  described  in 
this  assessment,  it  may  not  be  necessary  to  repeat  this  assessment  process.  However,  information 
from  a  previously  conducted  assessment  must  be  accessible  to  EQRO  reviewers. 

II.  ORIGIN  OF  THE  ASSESSMENT  TOOL 

A  number  of  public  and  private  sector  protocols  and  tools  were  examined  to  promote  consistency 
between  this  assessment  and  similar  public  and  private  sector  activities.  These  included:  1 )  the 
National  Committee  for  Quality  Assurance's  (NCQA)  1999  HEDIS®  publication:  Volume  5, 
HEDIS  Compliance  Audit™  Standards  and  Guidelines;  2)  tools  used  by  the  Island  Peer  Review 
Organization  (IPRO)  in  their  audits  of  HEDIS  measures;  and  3)  documents  from  the  MEDSTAT 
Group,  Inc.,  published  in  conjunction  with  work  performed  for  HCFA  in  1997  and  1998.  All 
these  protocols  include  an  assessment  of  the  structure  and  integrity  of  the  MCO's  underlying 


It  is  recognized  that  a  State  may  choose  an  organization  other  than  an  EQRO  as  defined  in  Federal 
regulation  to  conduct  as  assessment  of  MCO/PHP  Information  Systems  (IS).  However,  for  convenience,  in  this 
document  we  use  the  term  "external  quality  review  organization"  (EQRO)  to  refer  to  any  organization  conducting  an 
assessment  of  an  MCO/PHP  IS. 
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information  system,  and  of  the  MCO's  ability  to  collect  valid  data  from  a  variety  of  sources 
internal  and  external  to  the  organization. 


III.      OVERVIEW  OF  THE  ASSESSMENT 

Assessment  of  MCO/PHP  Information  Systems  is  a  four-step  process.  The  first  step  involves  the 
collection  of  standard  information  about  each  MCO  or  PHP's  IS.  This  is  accomplished  by  having 
the  MCO/  PHP  complete  an  Information  Systems  Capability  Assessment  (ISC A)  for  Managed 
Care  Organizations  and  Prepaid  Health  Plans.  The  ISCA  is  an  information  collection  tool 
provided  to  the  MCO/PHP  by  the  State  or  its  EQRO.  The  State  or  EQRO  will  define  a  time 
frame  in  which  the  MCP/PHP  is  expected  to  complete  and  return  the  tool.  Data  will  be  recorded 
on  the  tool  by  the  MCO/PHP.  Documents  from  the  MCO/PHP  are  also  requested  throughout  the 
tool,  and  are  summarized  on  the  checklist  at  the  end  of  this  assessment  tool.  These  are  to  be 
attached  to  the  tool  and  should  be  identified  as  applicable  to  the  numbered  item  on  the  tool  (e.g., 
II. B. 3,  or  IV. 6).  The  tool  itself  is  based  on  that  produced  by  MEDSTAT  Group,  Inc.,  with  some 
additional  elements  included  to  address  the  multiple  purposes  of  performing  assessments  of 
information  systems. 

The  second  and  third  steps  include  review  of  the  ISCA  information  by  EQRO  staff,  and 
interactive  sessions  with  MCO/PHP  staff  to:  1)  validate  the  ISCA,  and  2)  gather  additional 
information  to  assess  the  integrity  of  the  MCO/PHP's  information  system.  The  EQRO  will  use 
the  Managed  Care  Organization  Information  System  Review:  Worksheet  &  Interview  Guide  to 
conduct  interviews  with  MCO/PHP  staff  who  completed  the  ISCA,  as  well  as  other  necessary 
MCO/PHP  staff.  The  worksheet  is  an  annotated  version  of  the  ISCA  tool,  with  examples  of  the 
types  of  answers  a  reviewer  should  expect  to  receive,  along  with  additional  notes  relative  to  the 
issues  being  pursued.  During  an  onsite  visit,  EQRO  staff  can  explore  in  more  detail  the  responses 
submitted  by  the  MCO/PHP  on  the  ISCA.  The  space  to  record  answers  can  be  used  by  the 
reviewers  to  write  interview  notes,  or  document  specific  issues  identified  during  pre- visit 
analysis  that  need  to  be  pursued  with  the  MCO/PHP  during  the  onsite  visit. 

The  fourth  and  final  step  is  analyzing  the  findings  from  both  the  completed  Information  Systems 
Capability  Assessment  (ISCA)  for  Managed  Care  Organizations  and  Prepaid  Health  Plans,  and 
the  follow-up  discussions  with  MCO/PHP  staff. 


The  MCO/PHP  is  provided  a  copy  of  the  ISCA  (enclosed  in  the  following  section)  to  complete. 
Completing  the  ISCA  requires  that  the  MCO/PHP  provide  all  requested  documentation  identified 
on  a  checklist  at  the  end  of  the  assessment  tool,  and  return  it  to  the  EQRO  within  a  State-defined 


IV. 


ASSESSMENT  ACTIVITIES 


ACTIVITY  1: 


The  MCO/PHP  completes  the  Information  Systems  Capabilities 
Assessment  (ISCA) 
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time  frame.  The  EQRO  may  wish  to  provide  the  ISCA  in  an  electronic  format  to  the  MCO/PHP 
to  facilitate  completion  and  to  ensure  legibility  of  the  information  being  returned. 

ACTIVITY  2:  EQRO  review  of  the  completed  ISCA  and  accompanying  documents. 

The  EQRO  reviews  the  completed  ISCA  and  accompanying  documents,  assessing  the  adequacy 
of  MCO/PHP  policies  and  procedures  as  portrayed  by  the  information  submitted  by  the 
MCO/PHP  on  the  ISCA.  MCO/PHP  answers  should  be  evaluated  against  the  standards 
established  by  the  State  for:  1)  MCO/PHP  information  systems,  2)  calculating  and  reporting 
specific  plan-level  performance  measures,  and  3)  collecting  and  submitting  encounter  data  to  the 
State.  Where  an  answer  seems  incomplete,  or  indicates  an  inadequate  process,  the  EQRO  will 
note  that  section  for  follow-up  and  further  review  during  the  onsite  activities.  This  will  help  to 
focus  the  onsite  process  on  those  specific  policies,  procedures,  and  documentation  most  likely  to 
affect  the  integrity  of  information  collected  by  the  MCO. 

ACTIVITY  3:  Follow-up  interviews  with  MCO/  PHP  staff 

The  EQRO  reviewer(s)  conduct(s)  interviews  with  MCO/PHP  staff  responsible  for  completing 
the  ISCA,  as  well  as  additional  staff  responsible  for  aspects  of  the  MCO/PHP's  information 
systems  function.  The  interviews  focus  on  the  topics  outlined  in  the  structured  ISCA  interview 
guide,  with  additional  topics  covered  as  necessary  based  on  the  pre-onsite  analysis  of  the  ISCA. 

ACTIVITY  4:  Analysis  of  information  obtained  through  ISCA  and  follow-up 

interviews. 

The  EQRO  will  write  a  statement  of  findings  about  the  MCO/PHP's  information  systems  and  the 
implications  of  the  findings  for  the  following  activities,  as  directed  by  the  State: 

1 )  the  completeness  and  accuracy  of  any  encounter  data  collected  and  submitted  to  the  State; 

2)  calculation  or  validation  of  performance  measures; 

3)  the  ability  of  the  MCO/PHP  to  conduct  quality  assessment  and  improvement  initiatives;  and 

4)  the  ability  of  the  MCO/PHP  to  oversee  and  manage  the  delivery  of  health  care  to  its  enrollees. 
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Information  System  Capabilities  Assessment  (ISCA) 
for  Managed  Care  Organizations/Prepaid  Health  Plans 


I.      GENERAL  INFORMATION 


Please  provide  the  following  general  information: 

Note:  The  information  requested  below  pertains  to  the  collection  and  processing  of  data  for  an  MCO/PHP's 
Medicaid  line  of  business.  In  many  situations,  if  not  most,  this  may  be  no  different  than  how  an  MCO/PHP 
collects  and  processes  commercial  or  Medicare  data.  However,  for  questions  which  may  address  areas  where 
Medicaid  data  is  managed  differently  than  commercial  or  other  data,  please  provide  the  answers  to  the 
questions  as  they  relate  to  Medicaid  enrollees  and  Medicaid  data. 

A.     Contact  Information 

Please  insert  (or  verify  the  accuracy  of)  the  MCO/PHP  identification  information  below,  including 
the  MCO/PHP  name,  MCO/PHP  contact  name  and  title,  mailing  address,  telephone  and  fax 
numbers,  and  E-mail  address,  if  applicable. 


MCO/PHP 
Name: 

Contact  Name 
and  Title: 

Mailing 
address: 

Phone 
number: 

Fax  number: 

E-mail 

address: 

B.     Managed  Care  Model  Type  (Please  circle  one,  or  specify  "other".) 

MCO-staff  model  MCO-group  model  MCO-IPA  model  MCO-mixed  model 
Other  -  specify:  


C.     Year  Incorporated 


PHP 
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D.     Member  Enrollment  for  the  Last  Three  Years. 


Insurer 

Year  1: 

Year  2 

Year  3: 

i 

Privately  Insured 

Medicare 

Medicaid 

Other 

E.     Has  your  organization  ever  undergone  a  formal  information  systems  capability  assessment? 

Circle  a  response:  Yes  No 

If  yes,  who  performed  the  assessment?  

When  was  the  assessment  completed?  


NOTE:  If  your  MCO/PHP 's  information  system  has  been  formally  assessed  in  the  recent  past  (2  years  or  less), 
please  attach  a  copy  of  the  assessment  report.  Complete  only  those  sections  of  the  ISCA  that  are  not  covered  by 
or  have  changed  since  the  formal  assessment  was  conducted. 


4 


HCFA/CMSO/QPMG/DQSM  -  DRAFT  3/20/00 


Page  5 


Information  System  Capabilities  Assessment  (ISC A) 
INFORMATION  SYSTEMS:  DATA  PROCESSING  PROCEDURES  &  PERSONNEL 

1.     What  data  base  management  system(s)  (DBMS)  do/does  your  organization  use  to  store  Medicaid 
claims  and  encounter  data? 


2.     How  would  you  characterize  this/these  DBMSs?  (Circle  all  that  apply.) 


A.  Relational 

B.  Hierarchial 

C.  Indexed 

D.  Other 


E.  Network 

F.  Flat  File 

G.  Proprietary 

H.  Don't  Know 


3.     Into  what  DBMS(s),  if  any,  do  you  extract  relevant  Medicaid  encounter/claim/enrollment  detail  for 
analytic  reporting  purposes? 


4.     How  would  you  characterize  this/these  DBMS(s)?  (Circle  all  that  apply.) 

A.  Relational  E.  Network 

B.  Hierarchical  F.  Flat  File 

C.  Indexed  G.  Proprietary 

D.  Other  H.  Don't  Know 
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Information  System  Capabilities  Assessment  (ISC A) 


5.     What  programming  language(s)  do  your  programmers  use  to  create  Medicaid  data  extracts  or 

analytic  reports?  How  many  programmers  are  trained  and  capable  of  modifying  these  programs? 


6.     Do  you  calculate  defect  rates  for  programs? 

Circle  your  response.  Yes  No 
If  yes,  what  methods  do  you  use? 


What  was  the  most  recent  time  period? 
What  were  the  results? 


7.      Do  you  rely  on  any  quantitative  measures  of  programmer  performance?  If  so,  what  method(s)  do 
you  use? 
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Information  System  Capabilities  Assessment  (ISCA) 


8.     Approximately  what  percentage  of  your  organization's  programming  work  is  outsourced? 

% 


9.     What  is  the  average  experience,  in  years,  of  programmers  in  your  organization? 


1 0.    Approximately  how  much  resources  (time,  money)  is  spent  on  training  per  programmer  per  year? 
What  type  of  standard  training  for  programmers  is  provided?  What  type  of  additional  training  is 
provided? 


1 1 .    What  is  the  programmer  turnover  rate  for  each  of  the  last  3  years  (new  programmers  per  year/total 
programmers)? 

Year  1  (200x)  :  %  Year  2  (200x):  %  Year  3  (200x):  % 


1 2.    Outline  the  steps  of  the  maintenance  cycle  for  your  State's  mandated  Medicaid  reporting 

requirement(s).  Include  any  tasks  related  to  documentation,  debugging,  roll  out,  training,  etc.  The 
level  of  detail  should  result  in  10-25  steps  in  the  outline. 
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Information  System  Capabilities  Assessment  (ISCA) 


13.    What  is  the  process  for  version  control  when  code  is  revised? 


HCFA/CMSO/QPMG/DQSM  -  DRAFT  3/20/00 


Page  9 


Information  System  Capabilities  Assessment  (ISCA) 


How  does  your  organization  know  if  changes  to  the  claims/encounter/enrollment  tracking  system 
affect  required  reporting  to  the  State  Medicaid  program?  What  prompts  your  organization  to  changi 
these  systems? 


14. 


15.    Who  is  responsible  for  your  organization  meeting  the  State  Medicaid  reporting  requirements  (CEO, 
CFO,  COO)? 


16.  Staffing 

1 6a.  Describe  the  Medicaid  data  processing  organization  in  terms  of  staffing  and  their  expected 
productivity  goals.  What  is  the  overall  daily,  monthly  and  annual  productivity  of  overall 
department  and  by  processor? 


16b.  Describe  processor  training  from  new  hire  to  refresher  courses  for  seasoned  processors. 
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Information  System  Capabilities  Assessment  (ISCA) 

16c.  What  is  the  average  tenure  of  the  staff?  What  is  annual  turnover? 


17.  Security 

1 7a.  Describe  how  loss  of  Medicaid  claim  and  encounter  and  other  related  data  is  prevented  when 
systems  fail?  How  frequently  are  system  back-ups  performed?  Where  is  back-up  data  stored 


1 7b.  How  is  Medicaid  data  corruption  prevented  due  to  system  failure  or  to  program  error? 


17c.  Describe  the  controls  used  to  assure  all  Medicaid  claims  data  entered  into  the  system  is  fully 
accounted  for  (e.g.,  batch  control  sheets). 
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Information  System  Capabilities  Assessment  (ISCA) 


1  Id.  Describe  the  provisions  in  place  for  physical  security  of  the  computer  system  and  manual  files: 

m 


Premises 
Documents 
Computer  facilities 
Terminal  access  and  levels  of  security 


1 7e.  What  other  individuals  have  access  to  the  computer  system?  Customers?  Providers?  Describe 
their  access  and  the  security  that  is  maintained  restricting  or  controlling  such  access. 
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Information  System  Capabilities  Assessment  (ISC A) 


DATA  ACQUISITION  CAPABILITIES 

The  purpose  of  this  section  is  to  obtain  a  high-level  understanding  of  how  you  collect  and  maintain 
claims/encounters,  enrollment  information  and  data  on  ancillary  services  such  as  prescription  drugs. 

A.     Administrative  Data  (Claims  and  Encounter  Data) 

This  section  requests  information  on  input  data  sources  (e.g.,  paper  and  electronic  claims)  and  on  the 
transaction  system(s)  you  use. 


1 .     Do  you  use  standard  claims  or  encounter  forms  for  the  following?  If  yes,  please  specify  (e.g.,  HCFA 
1500,  UB  92) 


DATA 
SOURCE 

NO 

YES 

IF  YES,  PLEASE  SPECIFY 

Hospital 

Physician 

Drug 

Nursing  Home 

Home  Health 

Mental  health 

Other 

2.     We  would  like  to  understand  how  claims  or  encounters  are  submitted  to  your  plan.  We  are  also 
interested  in  an  estimate  of  what  percentage  (if  any)  of  services  provided  to  your  enrollees  by  all 
providers  serving  your  Medicaid  enrollees  are  NOT  submitted  as  claims  or  encounters  and  therefore 
are  not  represented  in  your  administrative  data.  Please  fill  in  the  following  table  with  the  appropriate 
percentages: 


CLAIMS  OR  ENCOUNTER  TYPES 


MEDIUM 

Hospital 

PCP 

Specialist 
Physician 

Dental 

Mental  health/ 
Substance  abuse 

Drug 

Other 

Claims/encounters  submitted  electronically 

Claims/encounters  submitted  on  paper 

Services  not  submitted  as  claims  or  encounters 

100°o 

1 00"  o 

1 00"  b 

1 00% 

100% 

100% 

1 00% 
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Information  System  Capabilities  Assessment  (ISCA) 


3.     Please  document  whether  the  following  data  elements  (data  fields)  are  required  for  each  of  the  types 


of  Medicaid  claims/encounters  identified  below.  If  required,  enter  an  "R"  in  the  appropriate  box. 


CLAIMS/ENCOUNTER  TYPES 


DATA  ELEMENTS 

Hospital 

Primary  Care 
Physician 

Specialist 
Physician 

Mental  Health/ 
Substance  Abuse 

Dental 

Drug 

other 

Patient  Gender 

Patient  DOB/ Age 

Diagnosis 

Procedure 

First  Date  of  Service 

Last  Date  of  Service 

Revenue  Code 

Provider  Specialty 

4.     How  many  diagnoses  and  procedures  are  captured  on  each  claim?  On  each  encounter? 


Claim 

Encounter 

Diagnoses 

Procedures 

Diagnoses 

Procedures 

Institutional  Data 

Professional  Data 

5a.    Can  you  distinguish  between  principal  and  secondary  diagnoses? 
Circle  your  response.       Yes  No 


5b.    If  "yes,"  to  5a,  above,  how  do  you  distinguish  between  principal  and  secondary  diagnoses? 


m 
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Information  System  Capabilities  Assessment  (ISCA) 


6.      Please  explain  what  happens  if  a  Medicaid  claim/encounter  is  submitted  and  one  or  more  required 
fields  are  missing,  incomplete  or  invalid.  For  example,  if  diagnosis  is  not  coded,  is  the  claims 
examiner  required  by  the  system  to  use  an  on-line  software  product  like  AutoCoder  to  determine  the 
correct  ICD-9  code? 

Institutional  Data: 


Professional  Data: 


7.     What  steps  do  you  take  to  verify  the  accuracy  of  submitted  information  (e.g.,  procedure  code- 
diagnosis  edits,  gender-diagnosis  edits,  gender-procedure  code  edits)? 

Institutional  Data: 


Professional  Data: 
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Information  System  Capabilities  Assessment  (ISCA) 
8.      Under  what  circumstances  can  claims  processors  change  Medicaid  claims/encounter  information? 


9.     Identify  any  instance  where  the  content  of  a  field  is  intentionally  different  from  the  description  or 
intended  use  of  the  field.  For  example,  if  the  dependent's  SSN  is  unknown,  do  you  enter  the 
member's  SSN  instead? 


10a.  How  are  Medicaid  claims/encounters  received? 


SOURCE 

Received  Directly  from  Provider 

Submitted  through  an  Intermediary 

Hospital 

Physician 

Pharmacy 

Nursing  Home 

Home  Health 

Mental  health 

Other 
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Information  System  Capabilities  Assessment  (ISCA) 


10b.  If  the  data  are  received  through  an  intermediary,  what  changes,  if  any,  are  made  to  the  data? 


1 1 .    Please  estimate  the  percentage  of  Medicaid  claims/encounters  that  are  coded  using  the  following 
coding  schemes: 


CODING  SCHEME 

Inpatient 
Diagnosis 

Inpatient 

Mr 

Procedure 

Ambulatory/ 
Outpatient 
Diagnosis 

Ambulatory/ 
Outpatient 
Procedure 

Drug 

ICD-9-CM 

CPT-4 

HCPCS 

Idsm-iv 

National  Drug  Code 

Internally  Developed 

Other  (specify) 

Not  required 

TOTAL 

100% 

100% 

100% 

100% 

100% 

1 2.    Please  identify  all  systems  through  which  service  and  utilization  data  for  the  Medicaid  population  are 
processed. 
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Information  System  Capabilities  Assessment  (ISCA) 

13.    Please  describe  any  major  systems  changes/updates  that  have  taken  place  in  the  last  three  years  in 
your  Medicaid  claims  or  encounter  system  (be  sure  to  provide  specific  dates  on  which  changes  were 
implemented). 

New  system  purchased  and  installed  to  replace  old  system. 

New  system  purchased  and  installed  to  replace  most  of  old  system;  old  system  still  used. 
Major  enhancements  to  old  system  (what  kinds  of  enhancements?). 
New  product  line  adjudicated  on  old  system. 
Conversion  of  a  product  line  from  one  system  to  another. 


14.  In  your  opinion,  have  any  of  these  changes  influenced,  even  temporarily,  the  quality  and/or  completeness 
of  the  Medicaid  data  that  are  collected?  If  so,  how  and  when? 


1 5.  How  many  years  of  Medicaid  data  are  retained  on-line?  How  is  historical  Medicaid  data  accessed 
when  needed? 
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Information  System  Capabilities  Assessment  (ISCA) 
16.  How  much  Medicaid  data  is  processed  on-line  vs.  batch?  If  batch,  how  often  are  they  run? 


1 7.  How  complete  are  the  Medicaid  data  three  months  after  the  close  of  the  reporting  period?  How  is 
completeness  estimated?  How  is  completeness  defined? 


18.  What  is  your  policy  regarding  Medicaid  claim/encounter  audits?  Are  Medicaid  encounters  audited 
regularly?  Randomly?  What  are  the  standards  regarding  timeliness  of  processing? 


19.  Please  provide  detail  on  system  edits  that  are  targeted  to  field  content,  consistency.  Are  diagnostic  and 
procedure  codes  edited  for  validity? 


HCFA/CMSO/QPMG/DQSM  -  DRAFT  3/20/00 


Page  19 


Information  System  Capabilities  Assessment  (ISCA) 


20.  Please  complete  the  following  table  for  Medicaid  claims  and  encounter  data  and  other  Medicaid 

administrative  data.  Provide  any  documentation  that  should  be  reviewed  to  explain  the  data  that  is  being 
submitted. 


Claims 

Encounters 

vainer 
Administrative 
Data 

Percent  of  total  service  volume 

Percent  complete 

How  are  the  above  statistics 
quantified? 

( 

V 

Incentives  for  data  submission 

2 1 .  Describe  the  Medicaid  claims/encounter  suspend  ("pend")  process  including  timeliness  of  reconciling 
pended  services. 
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Information  System  Capabilities  Assessment  (ISCA) 

4^  22.  Describe  how  Medicaid  claims  are  suspended/pended  for  medical  review,  for  non  approval  due  to 
tfp      missing  authorization  code(s)  or  for  other  reasons.  What  triggers  a  processor  to  follow  up  on  "pended" 
claims?  How  frequent  are  these  triggers? 


23.  If  any  Medicaid  services/providers  are  capitated,  have  you  performed  studies  on  the  completeness  of  the 
information  collected  on  capitated  services?  If  yes,  what  were  the  results? 


24a.     Identify  the  claim/encounter  system(s)  for  each  product  line  offered  to  Medicaid  enrollees.  (Note: 
Typically,  there  is  just  one  product  line  offered  to  Medicaid  enrollees,  but  there  may  be  some 
circumstances  in  which  a  MCO/PHP  offers  additional  product  lines  to  the  State;  e.g.,  S-CHIP,  partial 
risk  products. 


Systems  Used  to 
Process 

Product  Line: 

Product  Line: 

Product  Line: 

Fee-for-service 
(indemnity)  claims 

Capitated  service 
encounters 

Clinic  patient 
registrations 

Pharmacy  claims 

Other  (describe) 
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Information  System  Capabilities  Assessment  (ISCA) 


24b.     If  multiple  systems  are  used  to  process  claims  for  the  Medicaid  product  line,  document  how 

claims/encounters  are  ultimately  merged  into  Medicaid-specific  files-arid  on  which  platform?  Note  '^^^ 
which  merges  or  data  transfers  or  downloads  are  automated  and  which  rely  on  manual  processes. 


Are  these  merges  and/or  transfers  performed  in  batch?  With  what  frequency? 


24c.     Beginning  with  receipt  of  a  Medicaid  claim  in-house,  describe  the  claim  handling,  logging  and 
processes  that  precede  adjudication.  When  are  Medicaid  claims  assigned  a  document  control 
number  and  logged  or  scanned  into  the  system?  When  are  Medicaid  claims  microfilmed?  If  there  is 
a  delay  in  microfilming,  how  do  processors  access  a  claim  that  is  logged  into  the  system,  but  is  not 
yet  filmed? 
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Information  System  Capabilities  Assessment  (ISC A) 


4 


24d.     Please  provide  a  detailed  description  of  each  system  or  process  that  is  involved  in  adjudicating: 

•    A  professional  encounter(s)  for  a  capitated  service  (e.g.,  child  immunizations  that  arrive 
separately  from  the  office  visit.) 


A  hospital  claim  for  a  delivery  or  for  a  newborn  who  exceeds  its  mother's  stay. 


24e.     Discuss  which  decisions  in  processing  a  Medicaid  claim/encounter  are  automated,  which  are 

prompted  by  automated  messages  appearing  on  the  screen,  and  which  are  manual.  Document  the 
opportunities  a  processor  has  for  overriding  the  system  manually.  Is  there  a  report  documenting 
overrides  or  "exceptions"  generated  on  each  processor  and  reviewed  by  the  claim  supervisor?  Please 
describe  this  report. 


HCFA/CMSO/QPMG/DQSM  -  DRAFT  3/20/00 


Page  23 


Information  System  Capabilities  Assessment  (ISCA) 
24f.     Are  there  any  outside  parties  or  contractors  used  to  complete  adjudication,  including  but  not  limited 


to: 


□  Bill  auditors  (hospital  claims,  claims  over  a  certain  dollar  amount) 

□  Peer  or  medical  reviewers 

□  Sources  for  additional  charge  data  ("usual  &  customary") 

□  Bill  "repricing"  for  carved  out  benefits  (mental  health,  substance  abuse) 

How  is  this  data  incorporated  into  your  organization's  data? 


24g.     Describe  the  system's  editing  capabilities  that  assure  that  Medicaid  claims  are  adjudicated 

correctly.  Provide  a  list  of  the  specific  edits  that  are  performed  on  claims  as  they  are  adjudicated,  V. 
and  note:  1)  whether  the  edits  are  performed  pre-  or  post-payment;  and  2)  which  are  manual  and 
which  are  automated  functions. 


24h.     Discuss  the  routine  and  non-routine  (ad  hoc  or  special)  audits  that  are  performed  on 

claims/encounters  to  assure  the  quality  and  accuracy  and  timeliness  of  processing.  Note  which  audits 
are  performed  per  processor,  which  rely  on  targeted  samples  and  which  use  random  sampling 
techniques.  What  is  the  total  percentage  of  claims  on-hand  that  are  audited  through  these  QA 
processes?  How  frequently? 
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Information  System  Capabilities  Assessment  (ISC A) 


24i.     Please  describe  how  Medicaid  eligibility  files  are  updated,  how  frequently  and  who  has  "change" 
authority.  How  and  when  does  Medicaid  eligibility  verification  take  place? 


^^^24j.     How  are  encounters  for  capitated  services  handled  by  payment  functions?  What  message 
appears  to  notify  processors  that  they  are  handling  a  capitated  service? 


24k.     Desciibe  how  your  systems  and  procedures  handle  validation  and  payment  of  Medicaid  claims 
when  procedure  codes  are  not  provided. 
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Information  System  Capabilities  Assessment  (ISCA) 


241.     Where  does  the  system-generated  output  (EOBs,  letters,  etc.)  reside?  In-house?  In  a  separate 
facility?  If  located  elsewhere,  how  is  such  work  tracked  and  accounted  for? 


25a.     Describe  all  performance  monitoring  standards  for  Medicaid  claims/encounters  processing  and 
recent  actual  performance  results. 


^^^^^^^^^^  <• 

25b.     Describe  processor-specific  performance  goals  and  supervision  of  actual  vs.  target  performance.  Do 
processors  have  to  meet  goals  for  processing  speed?  Do  they  have  to  meet  goals  for  accuracy? 


25c.     How  is  performance  against  targets  figured  into  the  official  performance  appraisal  process?  Into 
processor  and  supervisor  compensation? 
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Information  System  Capabilities  Assessment  (ISCA) 


B.  Enrollment  System 


1 .  Please  describe  any  major  changes/updates  that  have  taken  place  in  the  last  three  years  in  your  Medicaid 
enrollment  data  system  (be  sure  to  identify  specific  dates  on  which  changes  were  implemented): 

□  New  enrollment  system  purchased  and  installed  to  replace  old  system. 

□  New  enrollment  system  purchased  and  installed  to  replace  most  of  old  system  -  old  system  still  used. 

□  Major  enhancements  to  old  system  (what  kinds  of  enhancements?). 

□  New  product  line  members  stored  on  old  system. 


2.  In  your  opinion,  have  any  of  these  changes  influenced,  even  temporarily,  the  quality  and/or  completeness 
of  the  Medicaid  data  that  are  collected?  If  so,  how  and  when? 


3.   How  does  your  plan  uniquely  identify  enrollees? 
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Information  System  Capabilities  Assessment  (ISCA) 


How  do  you  handle  enrollee  disenrollment  and  re-enrollment  in  the  Medicaid  product  line?  Does  the 
member  retain  the  same  ID?  £^ 


5.  Can  your  systems  track  enrollees  who  switch  from  one  product  line  (e.g.,  Medicaid,  commercial  plan, 
Medicare)  to  another? 

Circle  your  response.       Yes  No 


5a.  Can  you  track  an  enrollee's  initial  enrollment  date  with  your  MCO/PHP  or  is  a  new  enrollment  date 
assigned  when  a  member  enrolls  in  a  new  product  line? 


5b.  Can  you  track  previous  claim/encounter  data  or  are  you  unable  to  link  previous  claim/encounter  data 
across  product  lines? 


Under  what  circumstances,  if  any,  can  a  Medicaid  member  exist  under  more  than  one  identification 
number  within  your  MCO/PHP's  information  management  systems?  Under  what  circumstances,  if  any, 
can  a  member's  identification  number  change? 
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Information  System  Capabilities  Assessment  (ISC A) 
How  does  your  MCO/PHP  enroll  and  track  newborns  born  to  an  existing  Medicaid  enrollee  ? 


8.  If  your  MCO/PHP  has  a  Medicare  product  line,  describe  how  your  enrollment  systems  link  individuals 
simultaneously  enrolled  in  both  your  Medicare  product  line  and  the  Medicaid  plan  product  line. 


8a.  Is  claim/encounter  data  linked  for  Medicare/Medicaid  dual  eligibles  so  that  all  encounter  data  can  be 
identified  for  the  purposes  of  performance  measure  reporting? 


Circle  your  response.  Yes  No 


8b.  Is  claim/encounter  data  linked  for  individuals  enrolled  in  both  a  Medicare  and  Medicaid  plan  so  that  all 
encounter  data  can  be  identified  for  the  purposes  of  performance  measure  reporting? 

Circle  your  response.       Yes  No 
9.  How  often  is  Medicaid  enrollment  information  updated? 


10.  How  is  Medicaid  continuous  enrollment  being  defined?  In  particular,  does  your  system  have  any 
limitations  that  preclude  you  from  fully  implementing  continuous  enrollment  requirements  exactly  as 
specified  in  the  State  performance  measure  requirements? 
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Information  System  Capabilities  Assessment  (ISCA) 


1 1 .  Please  attach  a  copy  of  the  source  code  that  you  use  to  calculate  Medicaid  continuous  enrollment. 


12.  How  do  you  handle  breaks  in  Medicaid  enrollment~e.g.,  situations  where  a  Medicaid  enrollee  is 
disenrolled  one  day  and  reenrolled  the  next  simply  for  administrative  reasons?  Does  this  affect  your 
continuous  enrollment  calculations? 


13.  Do  you  have  restrictions  on  when  Medicaid  enrollees  can  enroll  or  disenroll?  Please  describe. 


14.  How  do  you  identify  and  count  Medicaid  member  months?  Medicaid  member  years? 


15.  Please  identify  all  data  from  which  claims/encounters  for  the  Medicaid  product  line  are  verified. 
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Information  System  Capabilities  Assessment  (ISC A) 


16.  Does  the  plan  offer  vision  or  pharmacy  benefits  to  its  Medicaid  members  that  are  different  from  the 
vision  or  pharmacy  benefits  offered  to  its  commercial  enrollees  (within  a  given  contract  or  market  area)? 
Circle  your  response.       Yes  No 

If  yes,  explain:  


16a.     If  vision  benefits  vary  by  benefit  package,  outline  the  different  options  available.  How  are  enrollees 
tracked? 


16b.     If  pharmacy  benefits  vary  by  benefit  package,  outline  the  different  options  available.  How  are 
enrollees  tracked? 
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Information  System  Capabilities  Assessment  (ISC A) 


C.  Ancillary  Systems 

Use  this  section  to  record  information  on  stand-alone  systems  or  benefits  provided  through  subcontracts,  such 
as  pharmacy  or  mental  health/substance  abuse. 


1 .  Does  your  MCO/PHP  incorporate  data  from  vendors  to  calculate  any  of  the  following  Medicaid  quality 
measures?  If  so,  which  measures  require  vendor  data? 

NOTE:  The  measures  listed  in  the  following  table  are  examples  of  measures  that  can  be  calculated  with 
administrative  data.  The  State  and  EQRO  should  tailor  this  table  to  list  those  measures  that  the  State 
requires  its  MCO/PHP  contractors  to  produce  and  any  other  measures  in  which  the  State  is  interested. 


MEASURE 

VENDOR  NAME 

Childhood  and  Adolescent  Immunization  Rate(s) 

Well  Child  Visits 

Initiation  of  Prenatal  Care 

Cervical  Cancer  Screening 

Chlamydia  Screening  in  Women 

Low  Birth  weight 

Prenatal  Care  in  First  Trimester 

 1 

Breast  Cancer  Screening  (Mammography 

Glycohemoglobin  Monitoring 

Ambulatory  Follow-Up  After  Hospitalization  For 
Specified  Mental  Health  Disorders 

Provider  Certification 

2.  Discuss  any  concerns  you  may  have  about  the  quality  or  completeness  of  any  vendor  data. 
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Information  System  Capabilities  Assessment  (ISCA) 

3.  Please  itemize  subcontracted  Medicaid  benefits  that  are  adjudicated  through  a  separate  system  that 
belongs  to  a  vendor. 


4.  Describe  the  kinds  of  information  sources  available  to  the  MCO/PHP  from  the  vendor  (e.g.,  monthly 
hard  copy  reports,  full  claims  data). 


5.  Do  you  evaluate  the  quality  of  this  information?  If  so,  how? 


6.   Did  you  incorporate  these  data  into  the  creation  of  Medicaid-related  studies?  If  not,  why  not? 
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Information  System  Capabilities  Assessment  (ISCA) 


D.  Integration  and  Control  of  Data  for  Performance  Measure  Reporting 

This  section  requests  information  on  how  your  MCO/PHP  integrates  Medicaid  claims,  encounter, 
membership,  provider,  vendor,  and  other  data  to  calculate  performance  rates.  All  questions  relate  to  your 
current  systems  and  processes,  unless  indicated  otherwise. 

File  Consolidation 

1 .  Please  attach  a  flowchart  outlining  the  structure  of  your  management  information  systems,  indicating 
data  integration  (i.e.,  claims  files,  encounter  files,  etc.).  For  an  example  of  the  minimum  level  of  detail 
requested,  please  refer  to  the  example  on  page  40.  Label  the  attachment  II. D.  1 . 

2.  In  consolidating  data  for  Medicaid  performance  measurement,  how  are  the  data  sets  for  each  measure 
collected: 

•  By  querying  the  processing  system  online? 

•  By  using  extract  files  created  for  analytical  purposes?  If  so,  how  frequently  are  the  files  updated? 
How  do  they  account  for  claim  and  encounter  submission  and  processing  lags?  How  is  the  file 
creation  process  checked  for  accuracy? 

•  By  using  a  separate  relational  database  or  data  warehouse  (i.e.,  a  performance  measure  repository)? 
If  so,  is  this  the  same  system  from  which  all  other  reporting  is  produced? 


3.   Describe  the  procedure  for  consolidating  Medicaid  claims  encounter,  member,  and  provider  data  for 
performance  measure  reporting  (whether  it  be  into  a  relational  database  or  file  extracts  on  a  measure  by 
measure  basis). 

3a.  How  many  different  sources  of  data  are  merged  together  to  create  reports? 


I 
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Information  System  Capabilities  Assessment  (ISCA) 

3b.  What  control  processes  are  in  place  to  ensure  data  merges  are  accurate  and  complete? 


3c.  What  control  processes  are  in  place  to  ensure  that  no  extraneous  data  are  captured  (e.g.,  lack  of 
specificity  in  patient  identifiers  may  lead  to  inclusion  of  non-eligible  members  or  to  double- 
counting)? 


3d.  Do  you  compare  samples  of  data  in  the  repository  to  transaction  files  to  verify  if  all  the  required 
data  are  captured  (e.g.,  were  any  members,  providers,  or  services  lost  in  the  process)? 


3e.  Describe  your  process(es)  to  monitor  that  the  required  level  of  coding  detail  is  maintained  (e.g.,  all 
significant  digits,  primary  and  secondary  diagnoses  remain)? 
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Information  System  Capabilities  Assessment  (ISCA) 


Describe  both  the  files  accessed  to  create  Medicaid  performance  measures  and  the  fields  from  those  files 
used  for  linking  or  analysis.  Use  either  a  schematic  or  text  to  respond. 


5.  Are  any  algorithms  used  to  check  the  reasonableness  of  data  integrated  to  report  Medicaid  performance 
measures? 


6.  Are  Medicaid  reports  created  from  a  vendor  software  product?  If  so,  how  frequently  are  the  files 
updated?  How  are  reports  checked  for  accuracy? 
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Information  System  Capabilities  Assessment  (ISCA) 

7.  Are  data  files  used  to  report  Medicaid  performance  measures  archived  and  labeled  with  the 
performance  period  in  question? 
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Information  System  Capabilities  Assessment  (ISC A) 


EXAMPLE 


Performance  Measure  Data:  Flowchart  of  Information  System  Structure 


Paper 
Claims/ 
Encounter 


Gairrfi/Encounter 
Transaction  System 


•System  edits 
•Pricing  files 
•Diagnosis 
procedure  files 
•  Provider  files 


Medical 
Records 


Plan-Level 

 ► 

Performance 

Measures 

* 
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Information  System  Capabilities  Assessment  (ISCA) 


Vendor  Data  Integration 

8.  Information  on  several  types  of  external  encounter  sources  is  requested.  In  the  table  on  the  following 
page,  for  each  type  of  delegated  service,  please  indicate  the  following: 

►  Second  column:  Indicate  the  number  of  vendors  contracted  (or  subcontracted)  to  provide  the 
Medicaid  service.  Include  vendors  that  offer  all  or  some  of  the  service. 

►  Third  column:  Indicate  whether  your  MCO/PHP  receives  member-level  data  for  any  Medicaid 
performance  measure  reporting  from  the  vendor(s).  Only  answer  "Yes"  if  all  data  received  from 
contracted  vendor(s)  are  at  the  member  level.  If  any  encounter-related  data  is  received  in  aggregate 
form,  you  should  answer  "No."  If  type  of  service  is  not  a  covered  benefit,  indicate  "N/A". 

►  Fourth  column:  Indicate  whether  all  data  needed  for  Medicaid  performance  measure  reporting  are 
integrated,  at  the  member-level,  with  MCO/PHP  administrative  data. 

►  Fifth  and  sixth  columns:  Rank  the  completeness  and  quality  of  the  Medicaid  data  provided  by  the 
vendor(s). 

Consider  data  received  from  all  sources  when  using  the  following  data  quality  grades: 
A  =  Data  are  complete  or  of  high  quality. 
B  =  Data  are  generally  complete  or  of  good  quality. 
C  =  Data  are  incomplete  or  of  poor  quality. 

►  In  the  seventh  column,  describe  any  concerns  you  have  in  ensuring  completeness  and  quality  of 
Medicaid  data  received  from  contracted  vendors.  If  measure  is  not  being  calculated  because  of  no 
eligible  members,  please  indicate  "N/A." 
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Information  System  Capabilities  Assessment  (ISCA) 


Performance  Measure  Repository  Structure 

If  your  MCO/PHP  uses  a  performance  measure  repository,  please  answer  the  following  question. 
Otherwise,  skip  to  the  Report  Production  section. 

9.  If  your  MCO/PHP  uses  a  performance  measure  repository  for  Medicaid  performance  measures, 
review  the  repository  structure.  Does  it  contain  all  the  key  information  necessary  for  Medicaid 
performance  measure  reporting? 


Report  Production 

10  .  Please  describe  your  Medicaid  report  production  logs  and  run  controls.  Please  describe  your 
Medicaid  performance  measure  report  generation  process. 

•>   


1 1 .    How  are  Medicaid  report  generation  programs  documented?  Is  there  a  type  of  version  control  in 
place? 
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12.    How  does  your  MCO/PHP  test  the  process  used  to  create  Medicaid  performance  measure  reports? 


13.    Are  Medicaid  performance  measure  reporting  programs  reviewed  by  supervisory  staff? 


14.    Do  you  have  internal  back-ups  for  performance  measure  programmers — i.e.,  do  others  know  the 
programming  language  and  the  structure  of  the  actual  programs?  Is  there  documentation? 


15.    How  are  revisions  to  Medicaid  claims,  encounters,  membership,  and  provider  data  systems 
managed? 
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PROVIDER  DATA 

Compensation  Structure 

The  purpose  of  this  section  is  to  evaluate  the  Medicaid  provider  compensation  structure,  as  this  may 
influence  the  quality  and  completeness  of  data.  Please  identify  the  percentage  of  member  months  in  your 
plan  contributed  by  Medicaid  members  whose  primary  care  providers  and  specialists  are  compensated 
through  each  of  the  following  payment  mechanisms. 


PAYMENT  MECHANISM 

Primary  Care  Physician 

Specialist  Physician 

1 .  Salaried 

2.     Fee-for-Service  -  no  withhold  or 
bonus 

1       Fee-for-Service  with  withhold 

+J  •               1  V-  V.     M  v/ 1      k_7     1   T  lvV>    TT  1111    TT  1111  11 VJ  1 VI 

Please  specify  %  withhold: 

4.     Fee-for-Service  with  bonus 
Bonus  range: 

5.     Capitated  -  no  withhold  or  bonus 

6.     Capitated  with  withhold 
Please  specify  %  withhold: 

7.     Capitated  with  bonus 
Bonus  range: 

8.  Other 

TOTAL 

100% 

100% 

9.      Please  describe  how  Medicaid  provider  directories  are  updated,  how  frequently,  and  who  has 
"change"  authority. 
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9a.  Does  your  MCO/PHP  maintain  provider  profiles  in  its  IS? 
Please  circle  response:     YES  NO 


9b.  If  yes  to  "a,"  what  provider  information  is  maintained  in  the  provider  profile  database;  e.g.  languages 
spoken,  special  accessibility  for  individuals  with  special  health  care  needs.  Other?  Please  describe: 


10.    How  are  Medicaid  fee  schedules  and  provider  compensation  rules  maintained?  Who  has  updating 
authority? 


1 1 .    Are  Medicaid  fee  schedules  and  contractual  payment  terms  automated?  Is  payment  against  the  schedules 
automated  for  all  types  of  participating  providers? 
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quested  Documentation 


e  documentation  requested  in  the  previous  questions  is  summarized  in  the  table  below.  Please  label  all 
attached  documentation  as  described  in  the  table,  and  when  applicable  by  the  item  number  from  the  ISCA  (e.g., 
III.B.  10).  Remember,  you  are  not  limited  to  providing  only  the  documentation  listed  below;  you  are  encouraged 
to  provide  any  additional  documentation  that  helps  clarify  an  answer  or  eliminates  the  need  for  a  lengthy 
response. 


Requested  Document 

Details 

Previous  Medicaid  Performance 
Measure  Audit  Reports 

Please  attach  final  reports  from  any  previous  Medicaid  performance 
measure  audits  in  which  your  MCO/PHP  participated  during  the 

Organizational  Chart 

Please  attach  an  organizational  chart  for  your  MCO/PHP.  The 

dial  I  ol  1UUIU  llIaKt  ClCal  U1C  1  ClallUllollip  al  llUIlg  Key 

individuals/departments  responsible  for  information  management, 
including  performance  measure  reporting. 

udid  iniegraLioii  now  v^iidii 

r  icd.sc  piuviue  a  liuwLiiari  nidi  gives  ait  overview  oi  tne  structure 
of  your  management  information  systems.  See  the  example 
provided  in  Section  II-D.  "Integration  and  Control  of  Data  for 
Performance  Measure  Reporting."  Be  sure  to  show  how  all  claims, 
encounter,  membership,  provider,  and  vendor  data  are  integrated 
for  performance  measure  reporting. 

Performance  Measure  Repository  File 
Structure  (if  applicable)  ] 

Provide  a  complete  file  structure,  file  format,  and  field  definitions 
for  the  performance  measure  repository. 

Program/Query  Language  for 
Performance  Measure  Repository 
Reporting  (if  applicable) 

Provide  full  documentation  on  the  software  programs  or  codes  used 
to  convert  performance  measure  repository  data  to  performance 
measures. 

Continuous  Enrollment  Source  Code 

Attach  a  copy  of  the  source  code/computer  programs  that  you  use 
to  calculate  continuous  enrollment  for  Medicaid  enrollees. 

Medicaid  Member  Months  Source 
Code 

Attach  a  copy  of  the  source  code/computer  programs  that  you  use 
to  calculate  member  months,  member  years  for  Medicaid  enrollees. 

Medicaid  Claims  Edits 

List  of  specific  edits  performed  on  claims  as  they  are  adjudicated 
with  notation  of  performance  timing  (pre-  or  post-payment)  and 
whether  they  are  manual  or  automated  functions. 

Statistics  on  Medicaid 
claims/encounters  and  other 
administrative  data 

Documentation  that  explains  statistics  reported  in  the  ISCA. 
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Reviewer  Worksheet  and  Interview  Guide 


This  annotated  version  of  the  Information  System  Capabilities  Assessment  for  Managed  Care  Organizations  / 
Prepaid  Health  Plans  (ISCA)  is  provided  for  EQRO  personnel  to:  1)  record  their  findings  from  the  review  of 
ISCA  forms  completed  by  the  each  MCO  /  PHP;  2)  note  issues  to  be  addressed  in  follow-up  interviews  with 
MCO  /PHP  personnel  and  3)  to  record  their  findings  from  those  interviews.  EQRO  staff  may  need  to  revise  this 
form  to  provide  additional  space  under  each  question  to  record  issues  and  findings. 

I.      GENERAL  INFORMATION 

Interviewee(s)  names  and  titles:  

Interviewer(s)  names  and  tiles:  

Date  of  interview:  


Please  provide  the  following  general  information: 

Note:  The  information  requested  below  pertains  to  the  collection  and  processing  of  data  for  an  MCO/PHP's 
Medicaid  line  of  business.  In  many  situations,  if  not  most,  this  may  be  no  different  than  how  an  MCO/PHP 
collects  and  processes  commercial  or  Medicare  data.  However,  for  questions  which  may  address  areas  where 
Medicaid  data  is  managed  differently  than  commercial  or  other  data,  please  provide  the  answers  to  the 
questions  as  they  relate  to  Medicaid  enrollees  and  Medicaid  data.  i 

A.     Contact  Information 


Please  insert  (or  verify  the  accuracy  of)  the  MCO/PHP  identification  information  below,  including 
the  MCO/PHP  name,  MCO/PHP  contact  name  and  title,  mailing  address,  telephone  and  fax 
numbers,  and  E-mail  address,  if  applicable. 


MCO/PHP  Name: 

Contact  Name  and  Title: 

Mailing  address: 

Phone  number: 

Fax  number 

E-mail  address: 
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B.     Managed  Care  Model  Type  (Please  circle  one,  or  specify  "other".) 

MCO-staff  model  MCO-group  model  MCO-IPA  MCO-mixed  model  PHP 
Other  -  specify:  


C.     Year  Incorporated 


D.     Member  Enrollment  for  the  Last  Three  Years. 


INSURER 

Year  1: 

Year  2: 

Year  3: 

Privately  Insured 

Medicare 

Medicaid 

Other 

E.     Has  your  organization  ever  undergone  a  formal  information  systems  capability  assessment? 

Circle  a  response:  Yes  No 

If  yes,  who  performed  the  assessment?  

When  was  the  assessment  completed?  

NOTE:  If  your  MCO/PHP's  information  system  has  been  formally  assessed  in  the  recent  past  (2  years  or  less), 
please  attach  a  copy  of  the  assessment  report.  Complete  only  those  sections  of  the  ISCA  that  are  not  covered  by 
or  have  changed  since  the  formal  assessment  was  conducted. 
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II.     INFORMATION  SYSTEMS:  DATA  PROCESSING  PROCEDURES  &  PERSONNEL 

Interviewee(s)  names  and  titles:  

Interviewer(s)  names  and  tiles:  

Date  of  interview:   


The  State  and  the  MCO/PHP  should  be  certain  that  data  being  reported  are  not  only  accurate  today;  but 
also  have  a  reasonable  chance  of  being  accurate  for  future  reporting  periods.  Future  accuracy  can  be 
predicted  by  assessing  the  MCO/PHP 's  systems  development  cycle  and  supporting  environment.  Plans 
that  lack  development  checkpoints  and  controls  are  much  more  likely  to  introduce  errors  as  systems 
change.  The  following  criteria  can  be  used  to  subjectively  assess  the  likelihood  of  future  reporting 
anomalies.  States  should  be  informed  that  very  few  programming  shops  in  the  world  really  meet  all  the 
desirable  criteria.  The  EQRO  will  consider  the  status  of  checkpoints  and  controls  in  its  overall 
assessment  of  findings. 

1 .     What  data  base  management  system(s)  (DBMS(s)  does  your  organization  use  to  store  Medicaid 
claims  and  encounter  data? 


2.     How  would  you  characterize  this/these  DBMSs?  Circle  all  that  apply. 


[Knowing  the  DBMS  provides  an  indication  of  the  organization  's  overall  level  of  sophistication.  Typical 
responses  would  include  Oracle,  DB2,  VTAM,  Paradox,  dBase,  R.  Base,  Sybase,  Informix,  SAS,  Rdb,  etc. J 

3.     Into  what  DBMS(s),  if  any,  do  you  extract  relevant  Medicaid  encounter/claim/enrollment  detail  for 
analytic  reporting  purposes? 


A.  Relational 

B.  Hierarchial 

C.  Indexed 

D.  Other 


E.  Network 

F.  Flat  File 


G.  Proprietary 

H.  Don't  Know 


HCFA/CMSO/QPMG/DQSM  -  DRAFT  3/20/00 


Page  48 


Information  System  Capabilities  Assessment 


Reviewer  Worksheet  and  Interview  Guide 


4.     How  would  you  characterize  this/these  DBMS(s)?  Circle  all  that  apply: 


A 
B. 
C. 
D 


Relational 
Hierarchical 
Indexed 
Other 


E.  Network 

F.  Flat  File 

G.  PROPRIETARY 

H.  Don't  Know 


[Answers  to  these  questions  will  provide  an  indicator  of  how  the  process  works.  Note  that  it  is  possible 
that  reports  are  generated  directly  from  the  incidence  database  without  any  intermediate  extraction.] 

5.     What  programming  language(s)  do  your  programmers  use  to  create  Medicaid  data  extracts  or 
analytic  reports?  How  many  programmers  are  trained  and  capable  of  modifying  these  programs? 

[For  example,  many  more  Cobol  programmers  are  available  on  the  market  than  for  Smalltalk] 


6.     Do  you  calculate  defect  rates  for  programs? 

CUlIo your  response.  Yes  No 
If  yes,  what  methods  do  you  use? 
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7.     Do  you  rely  on  any  quantitative  measures  of  programmer  performance?  If  so,  what  method(s)  do 
you  use? 


[Methods  to  calculate  defect  rates  and  productivity  measures  are  indicators  of  the  IS  organization 's  level 
of  sophistication.  Very  few  firms  calculate  either  of  these  very  well  today,  if  at  all.  Typical  methods 
would  include  Lines  of  Code  (LoC),  Pages  of  code,  ration  of  severe  bugs  to  all  bugs  found,  or  Function 
Points  (FP).J 


8.     Approximately  what  percentage  of  your  organization's  programming  work  is  outsourced? 

% 


9.     What  is  the  average  experience,  in  years,  of  programmers  in  your  organization? 


10.    Approximately  how  much  resources  (time,  money)  is  spent  on  training  per  programmer  per  year? 
What  type  of  standard  training  for  programmers  is  provided?  What  type  of  additional  training  is 

provided? 
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1 1 .    What  is  the  programmer  turnover  rate  for  each  of  the  last  3  years  (new  programmers  per  year/total 
programmers)? 

Year  1  (200x) :  %  Year  2  (200x):  %  Year  3  (200x):  % 


[These  questions  attempt  to  determine  the  stability  and  expertise  of  the  IS  department.  Answers  to  these 
questions  can  provide  additional  insight  into  the  development  cycle  responses.  Outsourcing  means  using 
non-employees  to  get  the  work  done,  sometimes  off-site,  in  which  case  project  specification,  management, 
coordination  and  acceptance  become  key  success  factors.  Ask  for  a  guess  if  the  turnover  rate  is  unknown. 
However,  not  knowing  the  rate  is  an  indicator  of  higher-than-usual  turnover.] 

12.    Outline  the  steps  of  the  maintenance  cycle  for  your  State's  mandated  Medicaid  reporting 

requirement(s).  Include  any  tasks  related  to  documentation,  debugging,  roll  out,  training,  etc.  The 
level  of  detail  should  result  in  10-25  steps  in  the  outline. 
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13.    What  is  the  process  for  version  control  when  code  is  revised? 

[The  IS  department  should  follow  a  standardized  process  when  updating  and  revising  code.  This  process 
should  include  safeguards  which  ensure  that  the  correct  version  of  a  program  is  in  use.] 


14.    How  does  your  organization  know  if  changes  to  the  claims/encounter/enrollment  tracking  system 
impact  required  reporting  to  the  State  Medicaid  program?  What  motivates  you  to  update  the 


[A  specific  individual  within  the  organization  should  be  responsible  for  determining  the  impact  of  any 
changes  made  to  the  plan 's  claims/encounter/enrollment  tracking  systems.  The  plan  should  have  in  place 
a  system  for  triggering  IS  staff  to  update  the  programs.] 


15.    Who  is  responsible  for  your  organization  meeting  the  State  Medicaid  reporting  requirements  (CEO, 


program ! 


CFO,  COO)'? 
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16.  Staffing 

1 6a.  Describe  the  data  processing  organization  in  terms  of  staffing  and  their  expected  productivity 
goals.  What  is  the  overall  daily,  monthly  and  annual  productivity  of  overall  department  and  by 
processor? 

[Unusually  high  productivity  goals  can  affect  the  accuracy  and  quality  of  a  processor  s  work.] 


16b.  Describe  processor  training  from  new  hire  to  refresher  courses  for  seasoned  processors. 

[New  hires  should  be  provided  with  on-the-job  training  and  supervision.  Supervisors  should  closely 
audit  the  work  of  new  hires  before  suspending  the  training  process.  Seasoned  processors  should  be 
given  occasional  refresher  courses  and  training  concerning  any  system  modifications.] 


16c.  What  is  the  average  tenure  of  the  staff?  What  is  annual  turnover? 

[A  larger  number  of  new  employees  or  high  turn-over  of  experienced  staff  could  result  in  decreased 
accuracy  and  processing  speed.] 


17.  Security 


1 7a.  Describe  how  loss  of  Medicaid  claim  and  encounter  and  other  related  data  is  prevented  when 
systems  fail?  How  frequently  are  system  back-ups  performed?  Where  is  back-up  data  stored? 
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[System  back-ups  should  be  performed  daily  (at  a  minimum)  to  prevent  against  data  loss.  Back-up 
data  should  be  stored  on  separate  systems  or  tape,  diskettes  or  DA  T,  and  stored  in  a  separate 
location  in  case  of  fire,  flood,  etc.] 


17b.  How  is  Medicaid  data  corruption  prevented  due  to  system  failure  or  to  program  error? 

[A  back-up  procedure  will  protect  the  data  from  destruction  due  to  system  failure  and  program 
error.  Plans  can  also  institute  additional  safeguards  to  protect  data  from  being  written  over  during 
these  processes.] 


1 7c.  Describe  the  controls  used  to  assure  all  Medicaid  claims  data  entered  into  the  system  is  fully 
accounted  for  (e.g.,  batch  control  sheets). 

[The  plan  should  have  a  process  in  place  which  ensures  that  all  claims/encounters  which  have  been 
logged  as  received  are  entered  into  the  system  and  processed.] 


1 7d.  Describe  the  provisions  in  place  for  physical  security  of  the  computer  system  and  manual  files: 

Premises 
•  Documents 

HCFA/CMSO/QPMG/DQSM  -  DRAFT  3/20/00  Page  54 


Information  System  Capabilities  Assessment  Reviewer  Worksheet  and  Interview  Guide 

•  Computer  facilities 

•  Terminal  access  and  levels  of  security 

[The  system  should  be  protected  from  both  unauthorized  usage  and  accidental  damage.  Paper 
based  claims/encounters  should  be  in  locked  storage  facilities  when  not  in  use.  The  computer  system 
and  terminals  should  be  protected  from  unauthorized  access  using  a  password  system  and  security 
screens.  Passwords  should  be  changed  frequently  and  should  be  re-set  whenever  an  employee 
terminates.] 
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1 7e.  What  other  individuals  have  access  to  the  computer  system?  Customers?  Providers?  Describe 
their  access  and  the  security  that  is  maintained  restricting  or  controlling  such  access.  j 

[Both  customers  and  providers  should  have  their  access  limited  to  read-only  so  that  they  cannot 
alter  any  files.  They  should  be  given  access  to  only  those  files  containing  their  own  patients  or 
members.  Customers  should  be  prevented  from  accessing  highly  confidential  patient  information  by 
being  given  "blinded" patient  names  and  "scrambled"  ID  numbers,  or  restricted  access  to 
particular  files.] 


m 
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DATA  ACQUISITION  CAPABILITIES 

interviewee(s)  names  and  titles:  

Interviewer(s)  names  and  tiles:  

Date  of  interview:  

The  purpose  of  this  section  is  to  obtain  a  high-level  understanding  of  how  you  collect  and  maintain 
claims/encounters,  enrollment  information  and  data  on  ancillary  services  such  as  prescription  drugs. 

A.    Administrative  Data  (Claims  and  Encounter  Data) 

This  section  requests  information  on  input  data  sources  (e.g.,  paper  and  electronic  claims)  and  on  the 
transaction  system(s)  you  use. 


1 .     Do  you  use  standard  claims  or  encounter  forms  for  the  following?  If  yes,  please  specify  (e.g.,  HCFA 
1500,  UB  92) 


DATA  SOURCE 


~1 


ospital 


hysician 


NO 


YES 


IF  YES,  PLEASE  SPECIFY 


Drug 
Other 


(Plans  that  do  not  use  either  HCFA  1500  or  UB  92  forms  may  be  using  forms  they  developed  themselves. 
If  a  plan  is  using  its  own  forms,  these  forms  should  be  reviewed  to  ensure  they  are  capturing  the  following 
key  data  elements:  patient  identification  information  [SSN,  name,  date  of  birth,  gender],  provider 
identifying  information  [Tax  ID,  name],  date  of  service,  place  of  service  and  diagnoses  and  procedure 
codes.  An  evaluation  of  their  forms  to  ascertain  adequacy  and  completeness  of  data  collection  may  be 
necessary . 


We  would  like  to  understand  the  means  by  which  claims  or  encounters  are  submitted  to  your  plan. 
We  are  also  interested  in  an  estimate  of  what  percentage  (if  any)  of  services  provided  to  your 
enrollees  by  all  providers  serving  your  Medicaid  enrollees  are  NOT  submitted  as  claims  or 
encounters  and  therefore  are  not  represented  in  your  administrative  data.  Please  fill  in  the  following 
table  with  the  appropriate  percentages: 
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CLAIMS  OR  ENCOUNTER  TYPES 


MEDIUM 

Hospital 

PCP 

Specialist 
Physician 

Dental 

Mental 
Health 
/Substance 
Abuse 

Drug 

Other 

Claims/encounters  submitted 
electronically 

Claims/encounters  submitted  on 
paper 

Services  not  submitted  as  claims 
or  encounters 

TOTAL 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

(Since  paper  forms  need  to  be  entered  into  a  plan 's  system,  processing  paper  forms  is  prone  to  error.  If  a 
plan  is  receiving  more  that  50  percent  of  its  data  on  paper  forms,  verify  the  data  checks  the  plan  uses  to 
test  processor  accuracy.  Electronic  data  submission  should  also  undergo  data  edits  and  validity  checks. 
Plans  with  a  high  percentage  of  unavailable  data  for  a  particular  category  will  have  difficulty  reporting 
measures  that  use  that  category.  For  example,  a  plan  receiving  no  drug  data  from  its  vendor  would  not  be 
able  to  report  the  HEDIS  measures  for  Outpatient  Drug  Utilization.) 
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3.      Please  document  whether  the  following  data  elements  are  required  for  each  of  the  types  of  Medicaid 
claims/encounters  identified  below.  If  required,  enter  an  "R"  in  the  appropriate  box. 

CLAIMS/ENCOUNTER  TYPES 


DATA  ELEMENTS 

Hospital 

Primary  Care 
Physician 

Specialist 
Physician 

Dental 

Mental 
Health  / 
Substance 
Abuse 

Drug 

other 

Patient  Gender 

Patient  DOB/Age 

Diagnosis 

Procedure 

First  Date  of  Service 

Last  Date  of  Service 

Revenue  Code 

Provider  Specialty 

(Standard  measures  of  plan  performance  such  as  Medicaid  HEDIS  are  dependent  upon  the 
availability  of  the  fields  listed  above.  If  procedure  codes  or  diagnosis  codes  are  not  available,  the 
data  will  not  include  the  necessary  level  of  detail  to  report  performance  measures.) 


4.     How  many  diagnoses  and  procedures  are  captured  on  each  claim?  On  each  encounter? 


Claim 

Encounter 

Diagnoses 

Procedures 

Diagnoses 

Procedures 

Institutional  Data 

Professional  Data 

(A  minimum  of  two  diagnosis  codes  and  two  procedure  codes  should  be  available.  If  only  one 
diagnosis  is  available,  it  may  be  difficult  to  identify  patients  with  chronic  conditions  such  as  diabetes 
or  asthma.) 


5a.    Can  you  distinguish  between  principal  and  secondary  diagnoses? 
Circle  your  response.       Yes  No 
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(Some  plans  will  consider  the  first  diagnosis  on  the  claim  to  be  principal.  Other  plans  determine  the 
principal  diagnosis  by  selecting  the  most  expensive  condition  represented. 

5b.    If  yes,  to  5. a  above  how  do  you  distinguish  between  principal  and  secondary  diagnoses? 


6.     Please  explain  what  happens  if  a  Medicaid  claim/encounter  is  submitted  and  one  or  more  required 
fields  are  missing,  incomplete  or  invalid.  For  example,  if  diagnosis  is  not  coded,  is  the  claims 
examiner  required  by  the  system  to  use  an  on-line  software  product  like  AutoCoder  to  determine  the 
correct  ICD-9  code? 

(The  use  of  an  automated  coding  product  such  as  GMIS'  AutoCoder  can  result  in  more  consistent  coding 
of  missing  information.  Plans  that  do  not  use  such  a  product  may  allow  processors  to  make  their  own 
decisions  on  appropriate  coding.  Processor  judgement  could  result  in  less  accurate  coding.) 

Institutional  Data: 


Professional  Data: 


7.     What  steps  do  you  take  to  verify  the  accuracy  of  submitted  information  (e.g.,  procedure  code- 
diagnosis  edits,  gender-diagnosis  edits,  gender-procedure  code  edits)? 

(For  example,  plans  will  often  verify  that  the  information  in  procedure  code  and  diagnosis  code  fields  are 
valid  codes.  Plans  may  also  verify  that  diagnosis  and  procedure  codes  are  appropriate  for  age  and 
gender.  For  example,  a. claim  with  a  procedure  of  hysterectomy  should  be  for  a  female  patient.) 
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Professional  Data: 


8.     Under  what  circumstances  can  claims  processors  change  Medicaid  claims/encounter  information? 

(If processors  are  given  the  ability  to  modify  claims/encounter  information,  the  accuracy  of  that 
information  could  be  affected  either  negatively  or  positively.  Processors  may  simply  correct  data  that  was 
submitted  incorrectly,  which  would  increase  the  quality  of  the  data.  However,  processors  may  also 
change  diagnosis  and  procedure  codes  which  could  result  in  a  loss  of  coding  specificity.  Does  the  plan 
check  processed  data  against  paper  claims?) 


9.      Identify  any  instance  where  the  content  of  a  field  is  intentionally  different  from  the  description  or 
intended  use  of  the  field.  For  example,  if  the  dependent's  SSN  is  unknown,  do  you  enter  the 
member's  SSN  instead? 

(Changing  the  content  of  afield  can  create  data  processing  issues.  For  example,  if  the  enrol  lee 's  SSN  is 
used  as  an  ID  for  a  number  of  dependents,  the  claim  may  be  given  the  age  and  sex  of  the  member  rather 
than  the  actual  patient.  The  use  of  the  enrollee 's  SSN  would  make  it  difficult  to  track  the  dependent 's 
experience  over  time.) 

•>   
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1  Oa.  How  are  Medicaid  claims/encounters  received? 


SOURCE 

Received  directly  from  provider 

Submitted  through  an  Intermediary 

Hospital 

Physician 

Pharmacy 

Other 

10b.  If  the  data  are  received  through  an  intermediary,  what  changes,  if  any,  are  made  to  the  data? 

(Intermediaries  that  are  processing  the  data,  such  as  a  pharmacy  benefit  firm,  could  modify  the  data, 
creating  a  data  set  that  is  inconsistent  with  the  plan 's  data.  The  intermediary  may  define  field  content 
differently  or  may  not  be  using  the  same  fields  as  the  plan,  making  it  difficult  to  integrate  the 
intermediary 's  data  into  the  plan 's  systems.  All  data  submitted  through  an  intermediary  should  be 
monitored  for  quality  by  the  plan.) 
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1 1 .    Please  estimate  the  percentage  of  Medicaid  claims/encounters  that  are  coded  using  the  following 
coding  schemes: 


CODING  SCHEME 

Inpatient 
Diagnosis 

Inpatient 
Procedure 

Ambulatory/ 
Outpatient 
Diagnosis 

Ambulatory/ 
Outpatient 
Procedure 

Drug 

ICD-9-CM 

CPT-4 

HCPCS 

DSM-IV 

National  Drug  Code 

Internally  Developed 

Other  (specify) 

Not  required 

TOTAL 

100% 

100% 

100% 

100% 

100% 

(If  a  plan  is  using  internally-developed  coding  schemes,  the  State  should  verify  whether  this  coding  can  be 
mapped  to  standard  coding  such  as  ICD-9  or  CPT-4.  If  the  coding  can  be  translated  for  reporting 
purposes  [Medicaid  HEDIS  requires  diagnosis  and  procedure  codes],  the  plan  should  provide 
information  on  the  level  of  specificity  with  which  the  coding  maps  to  standard  coding  [e.g.,  three-digit 
specificity  or  five-digit  specificity] .  If  the  mapping  has  a  low  level  of  specificity,  information  on  co- 
morbidities and  complications  may  not  be  retained  during  translation.) 


12.    Please  identify  all  systems  through  which  service  and  utilization  data  for  the  Medicaid  population  are 
processed. 

(Each  upgrade  or  consolidation  of  the  plan 's  information  system  has  the  potential  to  damage  the  quality 
of  the  data.  For  example,  data  could  be  lost  or  corrupted  during  a  system  conversion,  or  a  new  system 
could  limit  a  plan 's  access  to  historical  data.  Changes  in  data  quality  and  access  will  affect  the  plan 's 
ability  to  report  performance  measures  and  utilization.) 
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13.    Please  describe  any  major  systems  changes/updates  that  have  taken  place  in  the  last  three  years  in 
your  Medicaid  claims  or  encounter  system  (be  sure  to  provide  specific  dates  on  which  changes  wen 
implemented). 

New  system  purchased  and  installed  to  replace  old  system. 

•  New  system  purchased  and  installed  to  replace  most  of  old  system:  old  system  still  used. 
Major  enhancements  to  old  system  (what  kinds  of  enhancements?). 

•  New  product  line  adjudicated  on  old  system. 

Conversion  of  a  product  line  from  one  system  to  another. 

[When  a  plan  undertakes  any  major  system  changes  such  as  conversion  to  a  new  system,  the  system 
changes  could  affect  data  quality.  Data  quality  problems  include  corruption  of  data,  loss  of  data,  and 
loss  of  the  level  of  detail  within  the  data.  The  implementation  of  a  new  system  can  also  affect  the 
accessibility  of  historical  data.] 


14.  In  your  opinion,  have  any  of  these  changes  influenced,  even  temporarily,  the  quality  and/or  completeness 
of  the  Medicaid  data  that  are  collected,?  If  so,  how  and  when? 

[System  conversions  could  affect  the  quality  or  completeness  of  encounter  data  the  plan  submits  to  the  State, 
or  the  accuracy  of performance  measures.  A  temporaty  decrease  in  data  quality  could  be  a  sign  of  a  more 
serious  undiscovered  problem.] 
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15.  How  many  years  of  Medicaid  data  are  retained  on-line?  How  is  historical  Medicaid  data  accessed  when 
needed? 

[Due  to  system  constraints,  a  plan  may  remove  historical  data  and  place  it  in  off-line  storage.  The  MCO/ 
PHP  s  ability  to  report  on  experience  spanning  several  years  of  data  could  be  affected  by  the  accessibility 
of  the  data  stored  off-line.] 


16.  How  much  Medicaid  data  is  processed  on-line  vs.  batch?  If  batch,  how  often  are  they  run? 

[Data  which  are  processed  on-line  will  be  incorporated  into  the  system  on  a  real-time  basis.  If  batch 
processing  is  not  conducted  frequently,  it  can  result  in  data  processing  lags  which  affect  data 
completeness.] 


17.  How  complete  are  the  Medicaid  data  three  months  after  the  close  of  the  reporting  period?  How  is 
completeness  estimated?  How  is  completeness  defined? 

[The  comp'eteness  of  data  three  months  after  the  close  of  the  reporting  period  can  vary  greatly  by  plan.  A 
plan 's  contracting  arrangements  with  providers  can  affect  data  completeness.  Plans  that  delegate  provider 
payment  or  data  collection  to  medical  groups  or  IP  As  are  less  likely  to  have  complete  data  three  months 
after  the  reporting  period  ends.] 
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1 8.  What  is  your  policy  regarding  Medicaid  claim/encounter  audits?  Are  Medicaid  encounters  audited 
regularly?  Randomly?  What  are  the  standards  regarding  timeliness  of  processing? 

[Plans  should  be  performing  random  periodic  audits  of  their  encounter  data  to  determine  the  quality  of  data 
processing.  Plans  that  do  not  perform  audits  at  least  annually  are  not  closely  monitoring  the  quality  of  data 
processing.  Plan  standards  regarding  timeliness  of  processing  will  influence  the  lag  time  for  encounter 
data  processing.] 


19.  Please  provide  detail  on  system  edits  that  are  targeted  to  field  content,  consistency.  Are  diagnostic  and 
procedure  codes  edited  for  validity? 

{ 

[MCOs/PHPs  should  have  an  established,  standard  set  of  edits  that  verify  field  content  and  consistency. 
For  example,  a  field  content  data  edit  would  verify  that  a  valid  date  is  entered  into  the  date  of  service 
field.  Key  fields  which  should  be  edited  include  patient  identifying  information  (SSN,  name,  date  of 
birth,  sex),  provider  identifying  information  (name,  tax  ID,  type),  date  and  place  of  service,  and 
diagnosis  and  procedure  codes.  The  quality  of  diagnosis  and  procedure  coding  will  affect  the  validity  of 
reports  and  performance  measures  submitted  by  the  MCO/PHP.J 
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20.  Please  complete  the  following  table  for  Medicaid  claim  and  encounter  data  and  other  Medicaid 

administrative  data.  Provide  any  documentation  that  should  be  reviewed  to  explain  the  data  that  is  being 
submitted. 


Claims 

Encounters 

Other 
Administrative 
Data 

Pprrpnt  of*  total  cprvirp  vnliimp 

1  CI  tCUl  v/I  lUlal  9vl  TU1U111C 

Percent  complete 

How  are  the  above  statistics 
quantified? 

Incentives  for  data  submission 

[MCOs/PHPs  with  claims  data  comprising  more  than  50  percent  of  their  total  service  volume  are  likely 
to  have  a  more  complete  representation  of  total  MCO/PHP  experience  than  MCOs/PHPs  that  rely 
heavily  on  encounter  data.  While  providers  have  an  incentive  to  submit  claims  in  order  to  receive 
payment  for  services,  they  do  not  always  have  incentives  to  submit  encounter  information.  If  an 
MCO/PHP  does  not  offer  providers  an  incentive,  or  does  not  require  the  submission  of  encounter  data, 
the  MCO/PHP  may  not  receive  data  for  every  encounter.  Other  administrative  data  collected  by  an 
MCO/PHP  could  include  data  from  pharmacy  or  laboratory  vendors.] 

21.  Describe  the  Medicaid  claims/encounter  suspend  ("pend")  process  including  timeliness  of  reconciling 
pended  services. 

[Pended  claims/encounters  are  those  claims/encounters  that  have  been  suspended  during  processing 
because  they  failed  data  quality  edits  or  violated  provider  payment  parameters.  Information  on  these 
claims  and  encounters  will  not  be  available  for  reporting  until  they  have  been  reconciled  and  processed 
into  the  system.  Wlnat  percentage  of  claims  are  suspended  or  pended?] 
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22.  Describe  how  Medicaid  claims  are  suspended/pended  for  medical  review,  for  non  approval  due  to 
missing  authorization  code(s)  or  for  other  reasons.  What  triggers  a  processor  to  follow  up  on  "pended" 
claims?  How  frequent  are  these  triggers? 

[Review  and  processing  should  not  be  handled  by  the  same  employee.  A  system  should  be  in  place 
which  encourages  the  processor  to  follow-up  on  the  status  of  claims  in  review  that  have  not  yet  been 
approved  to  ensure  they  are  resolved.] 
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23.  If  any  Medicaid  services/providers  are  capitated,  have  you  performed  studies  on  the  completeness  of 
the  information  collected  on  capitated  services?  If  yes,  what  were  the  results? 


[Since  provider  payment  for  capitated  services  is  not  determined  by  the  encounter  data  submitted, 
providers  do  not  have  an  incentive  to  submit  complete  and  accurate  information  on  every  service 
provided.  Data  on  capitated  services  often  does  not  include  the  same  level  of  detail  as  fee-for-service 
claims  information.  Plans  should  be  aware  that  capitated  data  is  less  complete  and  should  audit  the 
data  at  least  annually  to  monitor  its  quality.] 


24a.     Identify  the  claim/encounter  system(s)  for  each  product  line  offered  to  Medicaid  enrollees: 


Systems  Used  to 
Process 

Product  Line: 

Product  Line: 

Product  Line: 

Fee-for-service 
(indemnity)  claims 

Capitated  service 
encounters 

Clinic  patient 
registrations 

Pharmacy  claims 

Other  (describe) 

24b.     If  multiple  systems  are  used  to  process  claims  for  the  Medicaid  product  line,  document  how 

claims/encounters  are  ultimately  merged  into  Medicaid-specific  files— and  on  which  platform?  Note 
which  merges  or  data  transfers  or  downloads  are  automated  and  which  rely  on  manual  processes. 

[When  data  are  merged  across  multiple  systems,  records  or  data  elements  can  be  altered  or  lost  during  the 
conversion  and  integration  processes.  Multiple  conversions,  integrations,  and  the  use  of  manual  processes 
will  increase  the  probability  of  an  error  occurring.] 
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Are  these  merges  and/or  transfers  performed  in  batch?  With  what  frequency? 

[Batch  processes  that  are  not  timely  can  result  in  data  processing  lags  which  affect  the  completeness  of  data 
after  the  close  of  the  reporting  period.] 


24c.     Beginning  with  receipt  of  a  Medicaid  claim  in-house,  describe  the  claim  handling,  logging  and 
processes  that  precede  adjudication.  When  are  Medicaid  claims  assigned  a  document  control 
number  and  logged  or  scanned  into  the  system?  When  are  Medicaid  claims  microfilmed?  If  there  is 
a  delay  in  microfilming,  how  do  processors  access  a  claim  that  is  logged  into  the  system,  but  is  not  ' 
yet  filmed? 
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24d.     Please  provide  a  detailed  description  of  each  system  or  process  that  is  involved  in  adjudicating: 

[Professional  encounters  arriving  separately  from  an  office  visit  may  not  be  processed  as  quickly  as  the 
actual  office  visits.  If  these  encounters  are  treated  as  "non-standard"  events,  the  plan  may  not  be  able  to 
easily  link  these  encounters  with  the  related  office  visit.  For  example,  newborns  exceeding  a  mother 's  stay 
may  have  their  hospital  stay  split  into  two  parts.  The  part  of  the  stay  which  coincides  with  the  mother 's 
hospitalization  may  be  processed  on  the  mother 's  claim  and  the  remainder  of  the  stay  could  be  processed 
separately.  Processing  the  newborn 's  stay  as  two  separate  claims  could  affect  the  plan 's  ability  to  report 
accurately  on  newborn  hospital  utilization.] 

A  professional  encounter(s)  for  a  capitated  service  (e.g.,  child  immunizations  that  arrive 
separately  from  the  office  visit.) 


A  hospital  claim  for  a  delivery  or  for  a  newborn  who  exceeds  its  mother's  stay. 
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24e.     Discuss  which  decisions  in  processing  a  Medicaid  claim/encounter  are  automated,  which  are  (4 
prompted  by  automated  messages  appearing  on  the  screen,  and  which  are  manual.  Document  the 
opportunities  a  processor  has  for  overriding  the  system  manually.  Is  there  a  report  documenting 
overrides  or  "exceptions"  generated  on  each  processor  and  reviewed  by  the  claim  supervisor?  Please 
describe  this  report. 

[If  processors  have  the  ability  to  override  the  system  manually  they  may  be  able  to  force  claims/encounters 
with  missing  information  through  the  system.  For  example,  a  processor  may  be  able  to  fill  in  missing 
diagnosis  or  procedures  codes.  Processors  could  also  use  override  codes  such  as  "99999"  or  "00000"  to 
fill  in  for  missing  codes.  If  the  system  does  not  "kick-out"  these  override  codes  during  processing,  the 
services  will  be  retained  in  the  system  without  diagnosis  or  procedure  detail.  Processors  may  also  be  able 
to  substitute  "000000000"  for  a  missing  SSN,  which  can  lead  to  services  for  unidentified  members  existing 
on  the  system.] 


24f.     Are  there  any  outside  parties  or  contractors  used  to  complete  adjudication,  including  but  not  limited 
to: 

□  Bill  auditors  (hospital  claims,  claims  over  a  certain  dollar  amount) 

□  Peer  or  medical  reviewers 

□  Sources  for  additional  charge  data  ("usual  &  customary") 

□  Bill  "repricing"  for  carved  out  benefits  (mental  health,  substance  abuse) 


How  is  this  data  incorporated  into  your  organization's  data? 
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[If  outside  parties  are  used,  the  plan  should  be  incorporating  data  generated  by  those  parties  into  the 
system.  The  data  should  first  be  run  through  the  plan 's  data  quality  checks  to  verify  its  accuracy  and 
completeness.] 


24g.     Describe  the  system's  editing  capabilities  that  assure  that  Medicaid  claims  are  adjudicated  correctly. 
Provide  a  list  of  the  specific  edits  that  are  performed  on  claims  as  they  are  adjudicated,  and  note:  1 ) 
whether  the  edits  are  performed  pre-  or  post-payment;  and  2)  which  are  manual  and  which  are 
automated  functions. 

[When  reviewing  plan  adjudication  edits,  the  State  should  concentrate  on  edits  which  affect  the  data  fields 
that  are  used  to  generate  plan  performance  measures  and  reports.  Are  outliers  for  length  of  stay  and 
charges  edited?  Utilizing  an  automated  editing  process  provides  more  consistent  results  that  do  not  require 
processor  judgment.  Edits  that  are  performed  pre-payment  can  prevent  invalid  data  from  being 
incorporated  into  the  system.] 


24h.     Discuss  the  routine  and  non-routine  (ad  hoc  or  special)  audits  that  are  performed  on 

claims/encounters  to  assure  the  quality  and  accuracy  and  timeliness  of  processing.  Note  which  audits 
arp  performed  per  processor,  which  rely  on  targeted  samples  and  which  use  random  sampling 
techniques.  What  is  the  total  percentage  of  claims  on-hand  luat  are  audited  through  these  QA 
processes?  How  frequently? 


[Note:  This  item  is  not  relevant  in  instances  where  the  EQRO  is  performing  encounter  data  validation. 
When  reviewing  edits  that  are  used  to  determine  processor  accuracy,  consider  that  these  edits  will  not 
provide  information  on  the  quality  of  the  initial  provider  data  submission.  The  audit  plan  should  include 
random  sampling  techniques  to  provide  an  overall  picture  of  quality.  Plans  will  often  concentrate  on 
auditing  complicated  or  aberrant  claims/encounters  rather  than  using  a  random  sample.  The  plan  should 
have  instituted  a  process  for  sharing  audit  results  with  the  processor  to  facilitate  quality  improvement.  / 
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24i.     Please  describe  how  Medicaid  eligibility  files  are  updated,  how  frequently  and  who  has  "change" 
authority.  How  and  when  does  Medicaid  eligibility  verification  take  place? 

[The  plan  should  add  new  enrollees  to  the  system  within  a  reasonable  amount  of  time  after  they  have 
enrolled.  Enrollees  should  not  be  experiencing  delays  in  access  to  care  due  to  plan  enrollment  processes. 
The  plan  may  be  using  a  different  enrollment  process  for  Medicaid  enrollees  than  for  enrollees  with 
commercial  coverage.] 


24j.     How  are  encounters  for  capitated  services  handled  by  payment  functions?  What  message  appears  to 
notify  processors  that  they  are  handling  a  capitated  service? 

[If  no  message  appears  to  notify  processors  that  they  are  handling  a  capitated  service,  these  services  could 
be  processed  incorrectly.  Payment  functions  can  be  suspended  or  modified  to  handle  capitated  services. 
The  plan  should  explain  how  capitated  services  are  processed  and  how  processing  affects  data  quality.] 
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• 


24k.     Describe  how  your  systems  and  procedures  handle  validation  and  payment  of  Medicaid  claims  when 
procedure  codes  are  not  provided. 

[Plans  requiring  valid  procedure  coding  for  all  claims/encounters  will  have  more  detailed  data  available 
for  reporting  and  analysis.  However,  these  plans  may  allow  processors  to  supply  missing  codes  using  a 
code  book  or  override  the  system  using  an  unspecified  code.  A  number  of  plans  use  programs  such  as  the 
GMIS  AutoCoder  product  to  fill  in  missing  codes.  When  a  plan  supplies  missing  codes,  the  coding  can  be 
less  accurate  than  codes  supplied  directly  by  the  provider  of  service.] 


241.     Where  does  the  system-generated  output  (EOBs,  letters,  etc.)  reside?  In-house?  In  a  separate 
facility?  If  located  elsewhere,  how  is  such  work  tracked  and  accounted  for? 

[Plans  that  have  delegated  the  production  of  EOBs,  letters  and  other  output  should  monitor  the  accuracy 
and  timeliness  of  those  activities.] 


25a.     Describe  all  performance  monitoring  standards  for  Medicaid  claims/encounters  processing  and 
recent  actual  performance  results. 
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25b.     Describe  processor-specific  performance  goals  and  supervision  of  actual  vs.  target  performance, 
processors  have  to  meet  goals  for  processing  speed?  Do  they  have  to  meet  goals  for  accuracy? 


25c.     How  is  performance  against  targets  figured  into  the  official  performance  appraisal  process?  Into 
processor  and  supervisor  compensation? 
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B.  Enrollment  System 

1 .  Please  describe  any  major  changes/updates  that  have  taken  place  in  the  last  three  years  in  your  Medicaid 
enrollment  data  system  (be  sure  to  identify  specific  dates  on  which  changes  were  implemented): 

□  New  enrollment  system  purchased  and  installed  to  replace  old  system. 

□  New  enrollment  system  purchased  and  installed  to  replace  most  of  old  system  -  old  system  still  used. 

□  Major  enhancements  to  old  system  (what  kinds  of  enhancements?). 

□  New  product  line  members  stored  on  old  system. 

[Changes  to  a  plan 's  enrollment  system  requiring  data  conversion  and  data  integration  can  create  data 
quality  problems.  Implementing  a  new  enrollment  system  could  lead  to  a  loss  of  access  to  data  on  the  old 
system,  or  the  assignment  of  new  member  numbers  for  all  enrollees.  Data  conversion  and  integration  can 
also  limit  a  plan 's  ability  to  track  an  enrollee 's  enrollment  history.  When  a  new  product  line  is  added  to  an 
existing  system,  a  plan  may  need  to  make  the  new  data  fit  the  older  process,  therefore  modifying  the  system 
to  "handle  "  new  information.  Implementing  such  modifications  can  be  difficult  for  a  plan  that  has  been 
using  the  same  system  for  a  number  of  years.  The  level  of  enrollment  detail  retained  can  be  affected  by  such 
modifications.] 


2.  In  your  opinion,  have  any  of  these  changes  influenced,  even  temporarily,  the  quality  and/or  completeness 
of  the  Medicaid  data  that  are  collected?  If  so,  how  and  when? 

[Consider  whether  changes  in  data  quality  will  affect  the  validity  of  the  data  submitted  to  the  State] 
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3.  How  does  your  plan  uniquely  identify  enrollees? 

(Major  changes  to  a  MCO/PHP 's  enrollment  system  could  involve  the  conversion  of  membership  data  to  a 
new  system.  When  MCO/PHPs  convert  members,  they  may  change  the  enrollee 's  ID  number,  making  it 
difficult  to  track  the  enrollee 's  enrollment  pattern  across  time.  Changes  to  the  enrollment  system  could  also 
lead  to  a  loss  of  data  for  specific  patients.) 


4.  How  do  you  handle  enrollee  disenrollment  and  re-enrollment  in  the  Medicaid  product  line?  Does  the 
member  retain  the  same  ID? 

(Enrollees  should  have  a  single  ID  number  to  facilitate  tracking  their  experience.  However,  some  plans 
change  an  enrollee 's  ID  number  when  the  enrollee  re-enrolls.  Experience  for  enrollees  who  have  switched 
ID  numbers  will  be  more  difficult  to  track.  Dependents  using  an  enrollee 's  ID  are  also  difficult  to  identify 
for  reporting  purposes.  For  example,  children  without  a  unique  ID  could  affect  the  ability  of  the  plan  to 
report  on  low  birth-weight  babies,  childhood  immunizations,  and  asthma  inpatient  admissions.  This  is  an  | 
important  point.  EQROs  should  give  higher  "grades  "  to  plans  which  have  good  methods  of  identifying 
enrollees.) 


5.  Can  your  systems  track  enrollees  who  switch  from  one  product  line  (e.g.,  Medicaid,  commercial  plan, 
Medicare)  to  another? 

Circle  your  response.       Yes  No 
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5a.  Can  you  track  an  enrollee's  initial  enrollment  date  with  your  MCO/PHP  or  is  a  new  enrollment  date 
assigned  when  an  enrollee  enrolls  in  a  new  product  line? 


5b.  Can  you  track  previous  claim/encounter  data  or  are  you  unable  to  link  previous  claim/encounter  data 
across  product  lines? 


Under  what  circumstances,  if  any,  can  a  Medicaid  member  exist  under  more  than  one  identification 
number  within  your  MCO/PHP's  information  management  systems?  Under  what  circumstances,  if  any, 
can  a  member's  identification  number  change? 


7.  How  does  your  MCO/PHP  enroll  and  track  newborns  born  to  an  existing  MCO/PHP  enrollee? 


8.   If  your  MCO/PHP  has  a  Medicare  product  line,  describe  how  your  enrollment  systems  link  individuals 
simultaneously  enrolled  in  both  your  Medicare  product  and  the  Medicaid  product  line. 


HCFA/CMSO/QPMG/DQSM  -  DRAFT  3/20/00 


Page  79 


Information  System  Capabilities  Assessment 


Reviewer  Worksheet  and  Interview  Guide 


8a.  Is  claim/encounter  data  linked  for  Medicare/Medicaid  dual  eligibles  so  that  all  encounter  data  can  be 
identified  for  the  purposes  of  performance  measure  reporting? 

Circle  your  response.  Yes  No 

8b.  Is  claim/encounter  data  linked  for  individuals  enrolled  in  both  a  Medicare  and  Medicaid  plan  so  that  all 
encounter  data  can  be  identified  for  the  purposes  of  performance  measure  reporting? 

Circle  your  response.  Yes  No 

9.  How  often  is  Medicaid  enrollment  information  updated? 
[Enrollment  information  should  be  updated  real-time,  daily,  or  weekly.] 


10.  How  is  Medicaid  continuous  enrollment  being  defined?  In  particular,  does  your  system  have  any 
limitations  that  preclude  you  from  fully  implementing  continuous  enrollment  requirements  exactly  as 
specified  in  the  State  performance  measure  requirements? 


1 1 .  Please  attach  a  copy  of  the  source  code  that  you  use  to  calculate  Medicaid  continuous  enrollment. 
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• 


12.  How  do  you  handle  breaks  in  Medicaid  enrollment~e.g.,  situations  where  a  Medicaid  enrollee  is 
disenrolled  one  day  and  reenrolled  the  next  simply  for  administrative  reasons?  Does  this  affect  your 
continuous  enrollment  calculations? 


13.  Do  you  have  restrictions  on  when  Medicaid  enrollees  can  enroll  or  disenroll?  Please  describe. 


14.  How  do  you  identify  and  count  Medicaid  member  months?  Medicaid  member  years? 


• 


15.  Please  identify  all  data  from  which  claims/encounters  for  the  Medicaid  product  line  are  verified. 

[Eligibility  of  the  patient  should  be  verified  before  claims  and  encounters  are  processed.  Dates  of 
enrollment  and  disenrollment  are  key  reporting  fields  for  Medicaid  HEDIS  measures.  Eligibility  status  for 
Medicaid  beneficiaries  is  dynamic  and  should  be  updated  frequently.  Eligibility  status  should  also  be 
verified  before  data  is  submitted  to  the  State] . 
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16.  Does  the  plan  offer  vision  or  pharmacy  benefits  to  its  Medicaid  members  that  are  different  from  the 
vision  or  pharmacy  benefits  offered  to  its  commercial  enrollees  (within  a  given  contract  or  market  area)^ 

Circle  your  response.       Yes  No 

1 6a.     If  vision  benefits  vary  by  benefit  package,  outline  the  different  options  available.  How  are  enrollees 
tracked? 


1 6b.     If  pharmacy  benefits  vary  by  benefit  package,  outline  the  different  options  available.  How  are 
enrollees  tracked? 


• 
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• 


.  Ancillary  Systems 


Tse  this  section  to  record  information  on  stand-alone  systems  or  benefits  provided  through  subcontracts,  such 
as  pharmacy  or  mental  health/substance  abuse. 


NOTE:  The  measures  listed  in  the  following  table  are  examples  of  measures  that  can  be  calculated  with 
administrative  data.  The  State  and  EQRO  should  tailor  this  table  to  list  those  measures  that  the  State 
requires  its  MCO/PHP  contractors  to  produce  and  any  other  measures  in  which  the  State  is  interested. 

1 .  Does  your  plan  incorporate  data  from  vendors  to  calculate  any  of  the  following  Medicaid  quality  measures? 
If  so,  which  measures  require  vendor  data? 


MEASURE 

VENDOR  NAME 

Childhood  and  Adolescent  Immunization  Rate(s) 

Well  Child  Visits 

Initiation  of  Prenatal  Care 

Cervical  Cancer  Screening 

Chlamydia  Screening  in  Women 

low  Birth- Weight 

Prenatal  Care  in  First  Trimester 

Breast  Cancer  Screening 

Glycohemoglobin  Monitoring 

Ambulatory  Follow-Up  After  Hospitalization  For 
Specified  Mental  Health  Disorders 

Provider  Certification 

(If  a  plan  is  using  vendor  data,  the  plan  should  have  a  formal  process  in  place  to  validate  that  data 
before  incorporating  it  into  their  information  systems.  The  plan  needs  to  check  the  vendor  data  for 
reliability,  completeness  and  timeliness  of  submission.) 

2.   Discuss  any  concerns  you  may  have  about  the  quality  or  completeness  of  any  vendor  data. 


(The  plan  should  have  staff  who  are  experienced  with  the  vendor 's  data.  Furthermore,  most  plans  will 
answer  this  question  by  saying  "we  have  no  concerns  ".  Probe  on  this  issue.  The  EQRO  should  "award 
points  "for  answers  demonstrating  understanding  of  potential  problems  with  vendor  data.) 
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3.  Please  itemize  subcontracted  Medicaid  benefits  that  are  adjudicated  through  a  separate  system  that 
belongs  to  a  vendor. 

[Many  plans  contract-out  services  for  pharmacy  benefits  management,  mental  health/substance  abuse, 
laboratory  and  radiology  services.  If  the  data  are  processed  on  the  vendor 's  system,  it  may  not  be 
forwarded  to  the  plan  in  a  complete  form  or  on  a  timely  basis.  Vendors  may  also  use  a  different  method  of 
processing  resulting  in  data  that  will  not  merge  with  or  complement  plan  data.] 


4.  Describe  the  kinds  of  information  sources  available  to  the  MCO/PHP  from  the  vendor  (e.g.,  monthly 
hard  copy  reports,  full  claims  data). 


5.  Do  you  evaluate  the  quality  of  this  information?  If  so,  how? 

[All  of  the  vendor  information  should  be  verified  for  accuracy  before  a  plan  loads  it  into  their  IS.  The  plan 
and  the  vendor  may  not  define  variables  consistently  or  use  the  same  reporting  format.] 
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Did  you  incorporate  these  data  into  the  creation  of  Medicaid-related  studies?  If  not,  why  not? 
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D.  Integration  and  Control  of  Data  for  Performance  Measure  Reporting 

This  section  requests  information  on  how  your  MCO/PHP  integrates  Medicaid  claims,  encounter, 
membership,  provider,  vendor,  and  other  data  to  calculate  performance  rates.  All  questions  relate  to  your 
current  systems  and  processes,  unless  indicated  otherwise. 

File  Consolidation 

1 .  Please  attach  a  flowchart  outlining  the  structure  of  your  management  information  systems,  indicating 
data  integration  (i.e.,  claims  files,  encounter  files,  etc.).  For  an  example  of  the  minimum  level  of  detail 
requested,  please  refer  to  the  example  on  page  92.  Label  the  attachment  II.D.  1 . 

2.  In  consolidating  data  for  Medicaid  performance  measurement,  how  are  the  data  sets  for  each  Medicaid 
measure  collected: 

♦  By  querying  the  processing  system  online? 

♦  By  using  extract  files  created  for  analytical  purposes?  If  so,  how  frequently  are  the  files  updated? 
How  do  they  account  for  claim  and  encounter  submission  and  processing  lags?  How  is  the  file 
creation  process  checked  for  accuracy? 

♦  By  using  a  separate  relational  database  or  data  warehouse  (i.e.,  a  performance  measure  repository)? 
If  so,  is  this  the  same  system  from  which  all  other  reporting  is  produced? 


3.   Describe  the  procedure  for  consolidating  claims/ encounter,  member,  and  provider  data  for  Medicaid 
performance  measure  reporting  (whether  it  be  into  a  relational  database  or  file  extracts  on  a  measure  by 
measure  basis). 

3a.  How  many  different  sources  of  data  are  merged  together  to  create  reports? 
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3b.  What  control  processes  are  in  place  to  ensure  data  merges  are  accurate  and  complete? 


3c.  What  control  processes  are  in  place  to  ensure  that  no  extraneous  data  are  captured  (e.g.,  lack  of 
specificity  in  patient  identifiers  may  lead  to  inclusion  of  non-eligible  members  or  to  double- 
counting)? 


3d.  Do  you  compare  samples  of  data  in  the  repository  to  transaction  files  to  verify  if  all  the  required  data 
are  captured  (e.g.,  were  any  members,  providers,  or  services  lost  in  the  process)? 


3e.  Describe  your  process(es)  to  monitor  that  the  required  level  of  coding  detail  is  maintained  (e.g.,  all 
significant  digits,  primary  and  secondary  diagnoses  remain)? 
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4.  Describe  both  the  files  accessed  to  create  Medicaid  performance  measures  and  the  fields  from  those  files^^ 
used  for  linking  or  analysis.  Use  either  a  schematic  or  text  to  respond.  ^0 


5.  Are  any  algorithms  used  to  check  the  reasonableness  of  data  integrated  to  report  Medicaid  performance 
measures? 


9 


6.  Are  Medicaid  reports  created  from  a  vendor  software  product?  If  so,  how  frequently  are  the  files 
updated?  How  are  reports  checked  for  accuracy? 


• 
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7.  Are  data  files  used  to  report  Medicaid  performance  measures  archived  and  labeled  with  the 
performance  period  in  question? 
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EXAM  PL 


Performance  Measure  Data:  Flowchart  of  Information  System  Structure 


Paper 
Claims/ 
Encounter 


Qaims/Encounter 
Transaction  System 

•  System  edits 

•  Pricing  files 

•  Diagnosis 
procedure  files 

•  Provider  files 


Medical 
Records 


Plan-Level 
Performance 
Measures 
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Vendor  Data  Integration 

8.   Information  on  several  types  of  external  encounter  sources  is  requested.  In  the  table  on  the  following 
page,  for  each  type  of  delegated  service,  please  indicate  the  following: 

►  Second  column:  Indicate  the  number  of  vendors  contracted  (or  subcontracted)  to  provide  the 
Medicaid  service.  Include  vendors  that  offer  all  or  some  of  the  service. 

►  Third  column:  Indicate  whether  your  MCO/PHP  receives  member-level  data  for  any  Medicaid 
performance  measure  reporting  from  the  vendor(s).  Only  answer  "Yes"  if  all  data  received  from 
contracted  vendor(s)  are  at  the  member  level.  If  any  encounter-related  data  is  received  in  aggregate 
form,  you  should  answer  "No."  If  type  of  service  is  not  a  covered  benefit,  indicate  "N/A". 

►  Fourth  column:  Indicate  whether  all  data  needed  for  Medicaid  performance  measure  reporting  are 
integrated,  at  the  member-level,  with  MCO/PHP  administrative  data. 

►  Fifth  and  sixth  columns:  Rank  the  completeness  and  quality  of  the  Medicaid  data  provided  by  the 
vendor(s). 

Consider  data  received  from  all  sources  when  using  the  following  data  quality  grades: 
A  =  Data  are  complete  or  of  high  quality. 
B  =  Data  are  generally  complete  or  of  good  quality. 
C  =  Data  are  incomplete  or  of  poor  quality. 

►  In  the  seventh  column,  describe  any  concerns  you  have  in  ensuring  completeness  and  quality  of 
Medicaid  data  received  from  contracted  vendors.  If  measure  is  not  being  calculated  because  of  no 
eligible  members,  please  indicate  "N/A." 
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Information  System  Capabilities  Assessment 


Reviewer  Worksheet  and  Interview  Guide 


Performance  Measure  Repository  Structure 

If  your  MCO/PHP  uses  a  performance  measure  repository,  please  answer  the  following  question. 
Otherwise,  skip  to  the  Report  Production  section. 

9.  If  your  MCO/PHP  uses  a  performance  measure  repository  for  Medicaid  performance  measures, 
review  the  repository  structure.  Does  it  contain  all  the  key  information  necessary  for  Medicaid 
performance  measure  reporting? 


Report  Production 

10  .  Please  describe  your  Medicaid  report  production  logs  and  run  controls.  Please  describe  your 
Medicaid  performance  measure  report  generation  process. 


1 1 .    How  are  Medicaid  report  generation  programs  documented?  Is  there  a  type  of  version  control  in 
place? 
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12.    How  does  your  MCO/PHP  test  the  process  used  to  create  Medicaid  performance  measure  reports? 


13.    Are  Medicaid  performance  measure  reporting  programs  reviewed  by  supervisory  staff? 


14.    Do  you  have  internal  back-ups  for  performance  measure  programmers — i.e.,  do  others  know  the 
programming  language  and  the  structure  of  the  actual  programs?  Is  there  documentation? 


15.    How  are  revisions  to  Medicaid  claims,  encounters,  membership,  and  provider  data  systems 
managed? 
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Information  System  Capabilities  Assessment 


Reviewer  Worksheet  and  Interview  Guide 


IV.    PROVIDER  DATA 

Compensation  Structure 

The  purpose  of  this  section  is  to  evaluate  the  Medicaid  provider  compensation  structure,  as  this  may 
influence  the  quality  and  completeness  of  data.  Please  identify  the  percentage  of  member  months  in  your 
plan  contributed  by  Medicaid  members  whose  primary  care  providers  and  specialists  are  compensated 
through  each  of  the  following  payment  mechanisms. 


PAYMENT  MECHANISM 

Primary  Care  Physician 

Specialist  Physician 

2.     Fee-for-Service  -  no  withhold  or 

hnmic 

~X        P1pA_fni*-Spt*vipp  with  withhold  - 

Please  specify  %  of  withhold: 

4.     Fee-for-Service  with  bonus 
Bonus  range: 

5.     Capitated  -  no  withhold  or  bonus 

^.     Capitated  with  withhold  -  Please 
\)      specify  %  of  withhold: 

7.     Capitated  with  bonus 
Bonus  range: 

8.  Other 

TOTAL 

100% 

100% 

[Timeliness  and  completeness  of  provider  data  submissions  often  varies  by  contracting  arrangement. 
Salaried  providers  work  directly  for  the  MCO/PHP  and  will  submit  data  on  a  timely  basis  if  data 
submission  is  a  parameter  in  their  contract  with  the  MCO/PHP.  Fee-for-service  providers  have  the 
largest  incentive  to  submit  accurate  and  complete  data  since  their  payment  depends  upon  it.  Capitated 
providers  will  need  incentives  to  submit  accurate  and  complete  data.  Their  compensation  should  be 
linked  to  data  submission,  which  can  be  done  through  the  use  of  bonuses  and  withholds.  For  example,  lag 
times  may  differ  by  compensation  arrangement  as  follows:  Capita  tion/Salaried-no  lag,  Fee-for-Service  - 
60  day  lag,  Hospital  -  45  day  lag.] 
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Please  describe  how  Medicaid  provider  directories  are  updated,  how  frequently,  and  who  has        |fl  W 

"change"  authority. 

[Provider  directories  should  be  updated  to  reflect  changes  in  provider  status  to  prevent  members 
from  selecting  providers  no  longer  under  contract  with  the  plan.  The  plan  should  have  adequate 
security  procedures  in  place  to  restrict  the  number  of  individuals  who  can  access  confidential 
provider  information  and  institute  changes  in  status.] 


9a.  Does  your  MCO/PHP  maintain  provider  profiles  in  its  IS? 
Please  circle  response:     YES  NO 


9b.  If  yes  to  "a,"  what  provider  information  is  maintained  in  the  provider  profile  database;  e.g.  languages 
spoken,  special  accessibility  for  individuals  with  special  health  care  needs.  Other?  Please  describe: 


10.    How  are  Medicaid  fee  schedules  and  provider  compensation  rules  maintained?  Who  has  updating 
authority? 

[Since  providers  consider  fee  schedule  and  compensation  information  to  be  confidential,  access  to 
this  information  should  be  restricted  by  the  MCO/PHP.  The  MCO/PHP  should  have  standardized 
process  for  updating  and  maintaining  this  information.] 
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*   

1 1 .    Are  Medicaid  fee  schedules  and  contractual  payment  terms  automated?  Is  payment  against  the 
schedules  automated  for  all  types  of  participating  providers? 

[Manual  payment  processes  are  more  prone  to  error  and  reduce  processing  speed.] 


Requested  Documentation 

The  documentation  requested  previously  is  summarized  in  the  table  below.  Please  label  all  attached 
documentation  as  described  in  the  table,  and  when  applicable  by  the  item  number  from  the  ISC  A  (e.g.,  III. B.  10). 
Remember,  you  are  not  limited  to  providing  only  the  documentation  listed  below;  you  are  encouraged  to  provide 
any  additional  documentation  that  helps  clarify  an  answer  or  eliminates  the  need  for  a  lengthy  response. 


^          Requested  Document 

Details 

Previous  Medicaid  Performance 
Measure  Audit  Reports 

Please  attach  final  reports  from  any  previous  Medicaid  performance 
measure  audits  in  which  your  MCO/PHP  participated  during  the 
past  two  years. 

Organizational  Chart 

Please  attach  an  organizational  chart  for  your  MCO/PHP.  The 
chart  should  make  clear  the  relationship  among  key 
individuals/departments  responsible  for  information  management 
including  performance  measure  reporting. 

Data  Integration  Flow  Chart 

Please  provide  a  flowchart  that  gives  an  overview  of  the  structure 
of  your  management  information  systems.  See  the  example 
provided  in  Section  II-D.  'Integration  and  Control  of  Data  for 
Performance  Measure  Reporting."  Be  sure  to  show  how  all  claims, 
encounter,  membership,  provider,  and  vendor  data  are  integrated 
for  performance  measure  reporting. 

Performance  Measure  Repository  File 
Structure  (if  applicable) 

Provide  a  complete  file  structure,  file  format,  and  field  definitions 
for  the  performance  measure  repository. 
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Program/Query  Language  for 
Performance  Measure  Repository 
Reporting  (if  applicable) 

Provide  full  documentation  on  the  software  programs  or  codes  usedi 
to  convert  performance  measure  repository  data  to  performance  ' 
measures. 

Continuous  Enrollment  Source  Code 

Attach  a  copy  of  the  source  code/computer  programs  that  you  use 
to  calculate  continuous  enrollment  for  Medicaid  enrollees. 

Medicaid  Member  Months  Source 
Code 

Attach  a  copy  of  the  source  code/computer  programs  that  you  use 
to  calculate  member  months  and  member  years  for  Medicaid 
enrollees. 

Medicaid  Claims  Edits 

List  of  specific  edits  performed  on  claims  as  they  are  adjudicated 
with  notation  of  performance  timing  (pre-  or  post-payment)  and 
whether  they  are  manual  or  automated  functions. 

Statistics  on  Medicaid 
claims/encounters  and  other 
administrative  data 

Documentation  that  explains  statistics  reported  in  the  ISCA. 
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The  Basics 

Anthony  J.  Asciutto 
The  MEDSTAT  Group 


Anthony  J.  Asciutto,  ma,  mph 
Biographical  Sketch 

(August  2000) 

Anthony  J.  Asciutto  is  a  Consultant  in  the  Chronic  Care  and  Disabilities  Unit  in  The  MEDSTAT  Group's 
Research  and  Policy  Division,  where  he  serves  as  an  internal  consultant  for  performance  measurement  and 
improvement  initiatives  and  Medicaid  managed  care  program  development  and  implementation.  Mr. 
Asciutto  currently  directs  a  HCFA-sponsored  evaluation  project,  "The  Evaluation  of  MassHealth  Quality 
Improvement  Plan,"  which  entails  the  documentation  and  assessment  of  the  approach  to  quality 
improvement  and  measurement  in  the  Medicaid  managed  care  programs  managed  by  the  Massachusetts 
Division  of  Medical  Assistance  (Medicaid).  He  also  provides  technical  and  programmatic  consultation  on  a 
number  of  provider  profiling  initiatives  for  state  Medicaid  programs.  He  recently  completed  the  "HCFA 
State  Health  Care  Reform  Monitoring  Project,"  where  he  led  the  development  of  Encounter  Data 
Workshops  for  state  Medicaid  Agencies  and  provided  technical  assistance  in  the  enhancement  of  a  national 
database  that  summarizes  the  operations  and  quality  management  initiatives  of  Medicaid  programs  across 
the  country.  In  addition,  he  has  assisted  in  the  conduct  of  HEDIS  compliance  audits  and  has  provided 
technical  expertise  in  the  development  of  report  cards  which  integrate  HEDIS  and  consumer  survey  data  to 
support  consumer  decision  making  and  performance  monitoring  activities  by  state  and  health  plan  officials. 

Prior  to  joining  The  MEDSTAT  Group,  Mr.  Asciutto  was  the  Director  of  Quality  Management  for  the 
Massachusetts  Division  of  Medical  Assistance  (Medicaid)  Primary  Care  Clinician  (PCC)  Plan.  In  this 
capacity,  he  oversaw  the  Quality  Management  Unit  for  the  state-administered  primary  care  case 
management  program,  which  covered  approximately  400,000  members.  He  was  responsible  for  agency- 
wide  clinical  quality  improvement  initiatives  that  evaluated  and  improved  processes  of  care,  service 
utilization,  health  outcomes  and  satisfaction  with  health  care  delivery.  He  was  also  responsible  for  federal 
government  reporting  requirements  for  managed  care  quality  management,  which  included  measurement 
design,  data  collection,  analysis  and  reporting  for  Medicaid  managed  care  programs,  including  contracted 
MCOs.  Mr.  Asciutto  led  the  agency's  annual  HEDIS,  Member  Survey  (CAHPS),  and  Encounter  Data 
initiatives  and  provided  measurement  and  performance  improvement  expertise  in  the  continued 
development  and  enhancement  of  the  PCC  Plan's  PCC  Profile  Report. 

Prior  to  joining  the  Massachusetts  Division  of  Medical  Assistance,  Mr.  Asciutto  was  a  Senior  Project 
Director  at  Harvard  Pilgrim  Health  Care  in  Boston,  MA,  where  he  was  a  member  of  the  management 
team  in  the  Clinical  Quality  Measurement  Unit  of  the  Medical  Director's  Office.  In  this  capacity  he 
provided  guidance  and  expertise  in  clinical  quality  measurement  and  research  methodology  and  directed 
cross-functional  research  teams  evaluating  the  quality  and  effectiveness  of  clinical  programs,  patient 
functioning,  service  utilization  and  satisfaction  with  care.  He  directed  an  extensive  evaluation  of 
corporate  and  local-level  Asthma  programs  to  inform  the  development  of  new  initiatives,  set  performance 
goals,  identify  best  practices  for  transfer  across  the  corporation,  monitor  progress  over  time,  and  prepare 
for  NCQA  accreditation  review. 

Mr.  Asciutto  has  extensive  experience  in  applied  research,  measurement,  and  consultation,  including  large- 
scale  survey  and  sampling  design,  data  collection,  analysis  and  reporting,  health  care  quality  management, 
applied  clinical  quality  measurement,  and  program  evaluation.  He  earned  his  MA  in  Applied  Economics, 
his  MPH  in  Public  Health  Policy  and  Administration,  and  his  AB  in  Psychology  from  the  University  of 
Michigan,  Ann  Arbor. 
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Marjorie  Porell 
Massachusetts  Division  of  Medical  Assistance 


• 


c 


Marjorie  Porell,  MS 


Ms.  Porell  has  held  several  positions  across  the  spectrum  of  health  care  settings  since 
receiving  a  Master's  of  Science  from  Carnegie  Mellon  University  in  1976.  In  her  current 
position,  as  assistant  director  in  the  Information  Analysis  Unit  at  the  Massachusetts 
Medicaid's  Division  of  Medical  Assistance,  Ms.  Porell  is  responsible  for  coordinating  all 
systems  resources  related  to  all  quality  of  care  initiatives.  She  is  also  the  lead  system 
contact  for  Massachusetts  in  the  area  of  encounter  data.  Ms.  Porell  has  been  with  the 
Massachusetts  Medicaid  Program  since  1994. 


• 


C0 

O 

PQ 

H 


03 

Q 

o 


o 


o 


o 
c 

» 

CO 

CO 
CO 

< 

"3 

o 


o 

c 
o 

CO 


CO 


H3 


C/3 


CO 

S3 

CO 

§  §  EC 


o 
C 

C 

o 
U 


o 

CO 
CO 


>  co 

*3 
o 


o 

Oh 


12  ^ 

-a  © 

s  o 

S  © 

co 
co 

CO  N 
CO 

<  £ 

£>  6 

§  Ph 


> 
O 


CD 

a 


C/3 

a 
PQ 

H 

i 
i 

a 
+-> 

Q 

<D 

§ 

o 

o 


go 
W 

00 
00 

W 
U 

GO  Q 

.  u 

Q  O 

U 

<< 

Q 


oo 

00 

W 

00 

<  W 
Q  ffi 

<  3 

GO  Q 

H 


os5 

Ph 

Oh 


Q 


oo 
W 
U 

P 
O 

GO 


< 
Q 


p 

< 


c3 


• 


t/3 

•  i-H 

in 

PQ 

H 

i 
i 

-4— > 

Q 

> 

o 

o 


u 

o 

H 


fin 

a 

a 
O 


CD 
O 

o 


O 

g 


CD 


o 


Oh  03 


C3  ^ 


o 
c3 


P  (ft 

0) 


d 

O 
O 

1)    C  O 


> 


0> 

o 

Oh 


a 

CO 
CO 

<D 

CO 
CO 


IZ3 

a 

OS 
CD 

H 

i 
i 

Q 

O 

o 


CO 
CO 

W 
U 

o 
o 

H 

U 

— 

o 

H 

«< 
Q 


H  a  «5 
C    S  O 

t>   W  Oh 


•    •  • 


% 


CD 


CD 
> 

o 

Ph 


t3 

P 

<3 


p 

CD 

OS 
CD 


C/3 

O 

'c/3 

PQ 

CD 

H 

i 
i 

Q 

+- > 

o 

o 


H 

-J 

< 


> 

s 

O  CO 

O  C  CO 
Oh    (5  O 


< 


Ph 


ccd 


H 


— 


— t  cd 
cd  H 
cd 

O  O 


00  Jh 
o 


CD  CD 

i— 1  £ 

P  OT  o 

rs>     Cd  ^ 


d 


CD 
CO 

CD 
5-h 

<h  Oh 


CD 
.d 


cd 

-a 

o 

•  l-H 

CD 

*d 
ft 

CD 
CD 

o 


CD 


cd 

a 
d 

cd 

•  T-H 

O 

•  T-H 

CD 

Oh 

CD 

^  CD 
cd  ^ 

d 
o 

CD 

cd 


o 

Oh 


d 
o 


cd  o 
cd  cd 


CD 

CD 


CD 
CD 

cd  O 

CD  5-i 

*d  Oh 


CD 

13 
a 
o 


bp  cd 

g  3 

r>  cd 
Q 

d 


d 
o 


CD 

CD 
.CD 
H— I 

O 

a  ft  ^ 
rj  >i  d 

T-H  CD 

cd  cd 

CD  IH 

pd   Ph  > 


CD 


CD 

CD 
CD 


CD 
CD 


cd 

CD 

,d 

O 

o 

CD 

CD 
ft 

S-l 

CD 


en  W 


B 
c 

3 

8 

c 

IU 


a 


■o  | 

v.  3 

55  5 


B 
c 

o 
o 
c 

Hi 


I 
n 

TO  0> 

55 


S 

o 


s 

o 


s 


s 

o 


8 
o 


s 

o 


8 

o 


8 

o 


TO 
> 

c 

CD 
if) 

I 


"O 

TO 
> 

c 

CD 
cn 

£ 


CD 
>. 
TO 
D. 

E 
-ft 
o 


la 
> 

C 

CD 
c/) 

I 


£ 

i 

73 


■D 
£ 
c 

.a> 

Q. 

o 


■o 
c 


c 

Q. 

o 
a: 


c 

a 

'o 
& 


8 


> 
c 

CD 
00 

I 


.10 

</> 

o 
c 
as 

TO 
T3 

>, 

TO 

E 


.to 
cn 
O 
c 

CD 
TO 

'•o 

>. 
TO 
T5 
C 
O 

o 
a> 


3 
T5 

8 
o 

Q. 

>, 
i— 
TO 

£ 


S 

0) 

o 
5 

E 
o 


TO 

> 
o3 


a> 


a> 

"D 

•5 
o 

a. 

CO 

c 
D 
5 

CD 


TO 
Q. 


U 
O 

C 
es 

.22  S 


< 

•5 

to 


o 

CO 
3 

o 

C3 

CO 
CO 


«4H 

o 

CD 

"3 


33  00 
> 


> 
O 

Oh 


£ 

GO 

CO 

a> 

CO 
GO 


C/3 

o 

•  i-H 

PQ 

H 

i 
i 

03 

> 

03 

Q 

o 

o 

a 

pq 


H 

< 


GO 
GO 

W 
H 
W 
-J 
Ph 


O 


a 

•  1— I 

> 
O 

o 

+-> 
a 

CO 

a> 

5— i 
Oh 
<D 
*-h 

+^ 

CO 


CO 

o 
o 

cs 

O 

o 


CO 

o 


CO 


B 

> 
O 


*  : 

S  g 


5/5 

in 

H 
— 


c/5 
O 

U 


O   ©  £ 

as 


o 
o 
o 


o 
o 
o 


o 
o 
o 


CO  CN 


oo 

ON 
OS 


b 
1 

£1 

6  I 


9 

e 
i- 


Os 
ON 


"5 


WdlAld  |s;oi 


m 


m 


»5 

= 

S3 
© 

I  § 

^  m 

o 
c 


o 

X 


f 

o 


r-  C\J 


oo 

OS 
ON 


Li  I 


91- 


Q 


6  | 

L 
9 


r-- 
o> 

ON 


3 


o  o  o  o  o 
o  o  o  o  o 
p  p  p  p  p  1 

lo       oo  cn  ^ 

|/\|dlAld  |e;oi 


tm 


a 

> 

H  * 


an 

Ctf  ! 


CM 


c3 

Q 

GO  £ 

CO  o 
GO  g 


o 


C3 

PS 


^  CZ3 

H  W  g 

2  H  8  1 

^  O 


o 
o 
p 

c\i 


o 
o 

LO 


o 
o 
o 


o 
o 

LO 


lAldlAld  leioi 


8 


■s 

GO 

On 
On 
On 


3 


•9 

CQ 


CO 

o 

6 


a 

> 
O 

PQ 


an 

I? 

C/3  g 


cd 


a  0 


© 


O  rB 
O 


^  o 


CO  CM 


OOOOO 
OOOOO 
LO     O     LO     O     LO  1 

csi  (N  ^  ^  6 

aiey  lAldlAlcdl  l^oi 


00 

0"\ 


B 

> 
O 


t/3 

o 


an 

cd  ! 


s 


03 

CO  £ 

go  a 

CO  ° 
CO  g 


cd 


CO 
CO 

W 
H 
W 


O 


S 

> 
O 

u 

JT1  (73 

H 


o 


is 


^  5C  S 

^  o  = 


3  « 

s 


c3 

'a 


w 


■S 

00 

On 
ON 
ON 


3 


•8 

& 

o 
B 

EC 

CO 

•ft 

O 

6 


£ 

> 
O 


an 

*  ! 

S 

GO  • 
CO 

go  o 


S-H 


H 
— 


co 

GO 

H 
— 


O 

u 


'a 


( 


.12  o 

O  0) 

C    (C  r 

■  MB           _  *■» 

(J           C  ■  — 


o  ■=  J2 

X  =  v> 
0  0 

00 
O) 

>- 

O 


LO 


CO 


CNJ 


0 

o 

Q_ 

CD 

Q£ 

0 

CO 


0 
c 

o 
o 
c 

LU 

□ 


oN 
O 
O 


O 
00 


O 
CD 


0s 

O 


O 
Csl 


O 


( 


c 

CO 


"t/5  co 

O  ^  0) 

E  co  c 
o  c  ■ - 

<i>  <  i: 
o  2  « 

I* 

00 

o> 

>- 

O 


£  5 
S  - 

>s  CD 
r-  CO 
CN  0) 

o  > 

*-  CO 

^  Q 

& 


o 
o 
c 
LU 


c 

CO 


CO 


LO 

CO 

CD 
CO 


"O 

CD 

n 
o 

CL 
CD 

M— 

CO 


& 
c 

o 
o 
c 

LU 

□ 


sO    ^0    vO  >^p 

o  o  o  o  o  o 

O  00  CD  CM 


m 


CD 
> 
O 

Vh 


S 

C/} 


c3 


t/3 

o 

•  1—1 

t/3 

PQ 

H 

i 
i 

+- » 

a 
Q 

a> 

a 

o 

o 

fl 


h-1  S 

^  s 


t/3 
CD 


C3 

o 

> 

03 
CD 


03 
X 


CO 

cd 

CO 
CO 
CD 
C 


CD 


a 
o 

J3 
u 

o 

•  ^ 

CD 


03  CD 


o 

03 

s 


rv  CD  co 

£  «  < 

03 


m 


a 

> 
O 

S  a 

h-H  PQ 


Q 


a 


CO 

co 


o 


Hie 

NH  « 

<  3 

<  2 


CO  O 


CO 

a 
o 


a 
o 
o 

o 


co 
0 
O 

» 

a 
o 

Oh 


CO 

o 

o 

D 
N 


u 

Q 


o 

a 
o 


03 

^CO 

ific 

*CO 

no 

•  • 

+^ 

CD 

iag 

~o 

• 

• 

u 

03 

03 

co 

O 

CD 

CO 

g  M 

a 


Sh  cd 
o 

03 

a 
o 

U 


03 
O 


CD 

s 

O 


*d 

CD 
O 

§ 

o 
o 

CD 

o 
£ 


co 
CD 

CO 
CO 


O 

o  33  ^ 

CD  '"O  •th 
N    ^  ^ 
O 


co 
=2  03 


03 

cr 

03 


£  *2  w 
Q 


•st- 
ir) 


£  "o 

(0  o 

o  o 

re  j_ 

o  «> 

>  = 

O  O 

m  LU 


o 

>- 

o 

Q.  a> 

CD  ' 

E 

0) 


CD 

Hill 

CN 

c\i 

0s 

CO 

CO 
CN 

m  [ 

CN 

0s* 
Is-; 

in 

00 

ill 

6s- 
LO 

CD 
CN 

CO 

CD 
< 


> 

CL 


CO 


CNJ 


o 
c 
0 

"D 

c 

CD 
Q_ 
CD 
"D 

o 

E 
0 

JZ 

O 

0s 


CD 


C\J 


fc 


s 

> 

o 

p..  a 

PQ 


a 


S 

CO 
CO 

<D 

CO 
CO 


Bo 

o 

o 


(5 


0) 

0 
o 

CO 

c 

c  < 

.2  a) 
15  .E 

N  > 
™  0) 

±r  o 

J  ©  oo 

£  0£  P 

^  </) 

II 

0 

O) 

re 

c 

a> 

<D 
Q. 


GO 
< 


CO 


CNJ 


p 

LO 


p 

O 
CNJ 


o 


O 


O 


O 


LO     O  LO 


TO 
03 
□ 

"D 
Q) 

■e 
o 

CD- 
CD 


B 

CD 
Q 

1— 

Q) 

C 

o 
u 
c 
LU 


CD  t- 


oo 


© 

pi: 

U 


o 
c 


2 
c 


X:  O 

Q.  N  P 
CO  =  > 

0  £  ° 

1  & 

I  o 
X 

I 

a 
■ 

o 

o 
u. 


o 
o 

CO 


o 
o 

C\J 


O 

o 


suo;;ez!|e;!dsoH 
jo  jeqairiN 


cl 
o 


Q. 

o 


D 


aj 
Q 

XD 
CD 

o 

Q- 

CD 


CD 
03 

Q 

i  

Q) 

o 
o 
c 
LU 


CD  -p 


Encounter  Data: 
Lessons  Learned  When  Using 
Encounter  Data  for  Monitoring 
Quality  in  Behavioral 
Health  Care 


Lessons  Learned  When  Using 
Encounter  Data  for  Monitoring 
Quality  in  Behavior  Health  Care 

James  J.  Linnane 
Connecticut  Department  of  Social  Services 


C/3 


GO 


S3 

o 

T3 

S-H 

S3 

d 
d 

a 

O 


IT)  Sh 

oo  5^ 
S 


o 


GO 
'Til 

<H-H  0> 

^  Oh 

•  i-H    1  1 

CD  O 


in 


CO 


CD 


O 
O 
O 

r 

O 


a-)  c3 
.5  o 

CD 

2  CD 

It 

«  Oh 


Ph 


CD 


X  o 

X  ^ 
.13  T3 

H  o 


CO 


PQ 

^H 


o 

rH 

GO 


sO 

O 
O 

m 

V 

Ph 

HH 

u 


o 

rv  X 

£  .2 

CO  > 
2  ^ 


(1  3 

CD 
O 


©  ^ 
©  go 
o  P 


u 

GO  -3 

"  s 

.IS  C/D 

H  p 

o  t 

Ji  pa 

C  2 

c  oo 

£  "3 

J  I 


Oh 

o 


o 
o 

e 

o 


GO 


o 

•  i—i 


CD 


o 

c 
W 

S-i 

u 

S3 

<t3 


b  8 

o  *g 

4  ^  73 
*  o 


•  i-H 

np 

o 

15 

4— > 

o 

o 

O 

u 

wi 

t5 

Oh 

a-) 

C/5 


5-H 


a-) 

fc! 
3 


Mm  U 

PQ    1  1 


CO 

8  m 


2  ui 


s 

CD 

CD 
+-> 

GO 
0 


<D 

GO 
GO 


CD 
CD 

CO 

O 

CO 

cd 
o 


O  o 

CD  T"H 

H  i 

CN 


GO 

u 
a 

>» 

CD 
O 


CD 
CD 

o 

13 
^— > 

c3 

S 


CO 

a 


CD 


CO 
•  1—1 

CD  g 
AC 

c3  x> 

pa 

o 
o 
c 

g  | 
<d£ 

+->  O 
O 

Cxi 

CD  39 


X 


c3 


CO 

EC 


o 
o 


as 


< 

GO 

o 


.SP 

'en 
> 

o 


o 
O 


GO 

CD 

> 

o 

CD 

> 
o 

Ph 


C3 

Q 

I 

o 

k 

Q 

O 

CO 

c3 


CO 

bo 

q  -c 
*  Si 

^  fa  ^ 

a&D  CO 


5~i 

bD 


u 

GO  U 

r/)  CD 

Q 

H  1 


CO  o 

T-1 


bD 

o 

+- > 

o 


o 
o 


bD  u 
o 

Ph 


o 

OJ  -d 

a-) 


< 

a 

<d 

C 

<d 

Oh 

<d 


2 
o 


CO 

■  i-H 

CO 

a 

<3 


CD 

CO 

CD 


O 
O 

U 


S 
o 

o 

(D 


o 

(D 


J  Ph 


CD 

CO 


3  _M   d  S 


S-H 

u 


(D 


CD 

CO 

CD 


43 

u 

•r~i  r 

u  u 


> 

•  H 
+- > 

GO 

•  1-H 

CD 


O 

C 

s* 
o 

U 

CD 

u 

CD 

a 


5  ^3 


<3 
o 

<D  ^3 
O 


(D 
+-> 

CO 
CO 

CD 

cj 
o 
o 
> 

co 
?-i 
CD 

•  H 

o 

Oh 

CO 

O 
+-> 

CO 

CD 
' — 1  CO 

*S  ° 

co 

CO 

CD  CD 

Q  l 

CD  ^D 


a3 

o 

> 

<D 

=s 
o 
o 
c 

o 

§ 

c 
c 


o 
o 
-a 
co  « 

CD  & 

CD'S 
^— »<: 

•4— >  ojq 


O 

CD 

CO 


o 
o 


CD 

•  i-H 
C    •  i-H 

O 


Q 

Oh 

W 

S3 

o 


5-1 

o 


C/3 

i 

c/) 


o 
c 

o 


CD 


X  M  u 


GO 

Q 

& 

o 
o 


o 

-  o 

0\  <u 


o 


5  ° 

c3  o 


in 

o 

GO 


cd 


GO 


O 

o 


CD 

0> 

a  O 

£  U 

42  Oh 

£  cd 

cd  cd 

a  to 


-j— > 

-a 

in 

O 

CD 

e 

o 

o 

o 
cd 

,£> 
o 

o 


QD 

I  S3 

<i)  o 


oo 

£ 

-2 

CD 
> 


CD 


03 


> 
Oi 

-t— > 

c 

O 
O 
S3 


CD      Oh  ■£ 

o 

03 


o 


o 

CD 

»— H  O0 

^  s 

|  y 


o 
o 
-a 

D 


o 
o 


< 
H 

Q 


o 

S-H 
-4-< 

a 
o 

ES  cd 

go 

2  s 
a  § 

o 
o 

c 

cd 


ON 

l-H 

•  «-h  CD 

VP  U 

■JB  u 


_  Cd 

o  ^ 

CD 
CD 


cr 

S-H 

CD 
> 
O 

GO 
CO 
CD 

CD 

5  CO 

a  I 


O 

> 


CO 
-4— » 

•  l-H 

CD 


«  l-H 

o 
o 

Oh 

O 
+-> 

CD 

CD 
• 1 — j 


Cd  '—I 

cd 

a 

CD 

> 

CD 


on 

ON 

<D 

^  fa 


CD 
•  i— i 

> 

s-l 

H 

< 


CD 
5-h 

=3 

CO 
CO 

CD 

Oh 

CD 
CO 

GO  § 


Q 


CO 

o  .2 

CO 

•  l-H 

CD  CT 

•  l-H  »"H 

f 

GO  I 

Q 


00 

C^  CO 

^>  CO 

O  CD 

£  g 
cd 


cd 


S 
o 

o 


C3 

O 

•  i-H 

o 

CD 
O 

^  cd 


CD 

o 

CD 


GO 

<D 
Oh 


00 


o 


o 

o  o 

O  CD 

c  ^ 

o  3 

O  cd 

oc  o 


z 


c3 


^3 


O 

o 

o 


o 


O 


o 


T3 

+-> 

o 

GO 


+-> 

GO 
GO 

<D 
+-> 

o 
4— > 

GO  GO 

Oh  T3 

GO  ^ 

£  2 

t3 


GO 

0) 
O 

o 
o 

i 

£ 

o 


GO 
■4— » 

fl 

GO 

<D 

o  s 

fa  > 
Oh  .3 


o 

c 
o 


<u 

■+- > 

c 

=5 

o 
o 
c 

c3 1 

o  & 
as  S 

g 

0>  o 
T3  » 

•  i-H  aj 

•  rH 

GO 

a 

o 

Oh  § 


v 


r 


oo 


<d 

s— > 

fl 
O 

•  i-H 

CD 

<D  ^ 

^  cd 
0>  CD 

43  ^ 

go  3 

4->  CD 

£  2 


CD 

a 

+— > 

CD 
+- » 

CD 

CD 
O 

a 


CD 


o 


o 
o 


o 

o 

GO  GO 
Oh 


o 


GO 


CD 


C-hH 

O  o« 

<D  CD 
GO  X) 

3 


CD  <D 
43  43 

GO  GO 


43  43 


O 
CD 
On 

o 
43 


5-1 

o 

+-> 

o 


cv 
^3 


So 


co 


CD 
43 

— > 

o 
Q 


S-H 

o 
"> 

43 

!- 
d> 
-I— > 

55 
=3 
O 
O 

c 

o 

45 

C 

£5 


o 
o 

hh 


o 
o 


OS 


O 

CD 

D 

U 

GO 

<D 
O 

o 
X 


CD 

o 

Oh 

eD 

o 

-a 
o 

cd 


CO 
3 


3 

o 

o 

o 

£  5 

03  <D 


CO 

o 


cd 
+-> 

S3 

o 

•  I 

5-h 


<D 

co  cd 


2 


o 

«  o 


CO 


(D 
^— > 

§ 

O 

o 

a 

o  g 

CO  ^ 

O  b 

»  03 


-+-> 


<D 

o 

CO 


00 

Q 


GO 

HH 


CD 

o 

•  1— I 


CD 


CD 


OH  ^ 


o 

o 

73 


Plh 

I  I 


a 

o 

CD 

CQ 


co  ^3 
M 


5-H 

03 


O 


o 
£  a. 

c  .2 

CD  & 

-2  1?  « 

03    n  <D 


- 


( 


o 


GO 

o 


o 

Oh 

0) 

S-h 

4— > 

CD 

u 


o 
o 
o 

(N 
43 

o 

S3 


c3 


O 
O 
O 
(N 

43 
O 


<3 


GO 

S3 

o 

a 


a-) 

13 


S  GO 

o  o 


C\  OC 

•  • 

^  O 


o 


GO 

a 

CD 


Q 


o 

cd 
O 


o 
o 


o 

o 

+-> 
o 

I 


»  H 

a-) 

•  r— I 

d 


c3 
O 

> 
cd 

O 

u 


<D 

GO 

<D 

•  1— I 

o 

Oh 

o 
o 


o 

o  a 

CD 


C3 


43 
o 

a 

CD 

GO 

<D 
O 

C 

o 


ts 


GO 
C 

•  i-H 

GO 


4— > 

o 
H 


o3 
J-* 
O 

*  i— 

<u 

J-l 

<D 
-i— > 

C 

o 
o 
c 

M— l 

o 

c 

c 
c 


o 
o 

T3 


o 
o 


ON 


o 


c3 


Oh 
44 

s 

43 
a 

PQ 


o 

in 
O 

GO 


o 

a  ° 


o 


C/3 

o 


o 
o 

c3 


c3 
O 

> 

43 

<D 
42 

T3 

> 

o 

CD 

o 
a 

>  C 

4^ 

,  H 
O  ^3 

sO  £J 
^  43 


43 

o 

^  fa 

H-H 


CO 

a-) 

a) 
-j— » 


00 


o 

c 

> 


03 


O     H-H  fa 


a-) 


C/3 

C3 
O 

o 

03 
43 

15 


43 
o 

o 

43 

00 


Q 

B  ^ 

u  ^ 


2 
O 

o 
13 


43 

o 
a-> 

o 

X 

03  ^— ^ 

S03 

tS.2 

O  4=) 


as 
i- 
O 

4= 
42 

-t-> 

c 

O 
G 

CD 

o 

4) 

S 

c 


o 
o 


o 
o 


ON 


r 


03 


Q 


o 


CD 


-M >  03 

s  £ 

•  1— I  ^> 

§  s 

o 

03 


c 


CD 

o 

GO 

j2  £ 

03 

O  ctf 

^  o 

+-  > 


CD 
O 

"S3 
o 


•  i— i 
O 

Cm 

GO 

a 


^  s 


o3 
PS 

a 

C+H 

o 
o 

o 

a 


a 
o 
o 

GO 

O 
O 

0) 


g 

"3 

H 


CD 
M 


a>  P 

2  3 
^  g 


o 

en 


o 
o 

O  g 
Q  8 

s  - 

_«  cn 

s  £ 


CD 

> 

CD 


p 

CD 
O 

IS 

P 

>^ 

-M 

•  T— ( 
P 


o 
o 


p 

CD 


c3 


GO 


O 
ft 

ft 

44 


1 

CD 


o 
o 


P    c/3  ^ 


CD 
M 


a3 
O 

'> 
a3 

<D 

C 
O 

a 
c 

32 

c 

03 

c 


o 
o 

!— 


o 
o 


CO 


Q 

J-l 


O 

o 


CD 


<D 

e 

•  T— I 

CD 

^3 


CO 

O 
C 

CD 


o 

CD 


<D 
^— > 

-a 

s 

-a 

as  ^ 

&  § 

43 

o 


c3  cS 

43  43 


CD 
go 

•  i— H      •  -H 

-4— »  -4— > 

cd  cd 
nd 


S-H 

43 

o 

CO 


CD 
CD 


CD 
42 

CD 

CO 

t 

<D 

o  2 

43  co 

CO  co 

S3  t3 

*4— >  C3 


3 


43 

o 

33 


S3 

(D 

c  a- 

^  CO 

o  -g 

K  CO 


cd 


O 


CO 

H 

U 


in 

<D 
+^ 

cd 
On! 


C/3 

cd 
O 

m 


\0 

IT) 

o 

•  H 


CD 


CO 


CD 

Oh 

-S3 
-a 

CD 

CD 
CD 


S3 

1— H 

o 


E3 

U  Q 


> 

x: 
a 
,o 

5— 

c 

O 

o 
c 

<u 

M-H 

o 
c 

G 
G 


CO       >  3 


u 
o 


03 

GO 


O 

o 


On 


*  CO 
GO 

CD 

S 

o 

CD 

PQ 


GO 

£ 

•  1— I 
O 

GO 

O 

GO 

O 


o 


Q 

W 


5-H 


<D 
o 

GO 


CD 

a  & 

U  g 

.9  I 

GO  ^ 
+3  CD 


173 

(D 
CO 
CO 

Oh 

CD 
CD 


CD 
CO 
CD 

4— > 

CD 

X 
CD 

CD 

o 

GO 


GO  0« 

•  H 


CD 
O 

O 
O 


GO 
4— » 

CD 

o 


CD 

*5 

c 

O 
O 

GO 


S3 


O 

o 


o 


CD 


GO 


CD 

■  rH 
4— » 

Oh 


|£ 

•  •  a 

ft  Ofi 

d  <u 

C/3  X) 

s  * 

d  2 

03  4-> 

O  g  ^ 


o 

GO 

CD 

GO 

cd 

CD 


5^ 

CD 
4— > 
^— • 

CD 

•a  u 

o  q 


^  O 


Q 


GO 


GO 
> 

CD 


CD 

o 


CD 
CO 

CD 


a  s 

o 


C3 


CO 
CO 

o 

O 


O 

>  2 


o 

5-1 


c3 
3 


•a  ^ 

•  r-H  O 

GO 

a  s 

8  | 


O 

GO 
O 

ta 
o 


co  >-> 


GO 

& 

o 
C 

0-> 


o 
c 

CD 


o 

o 

GO 

> 
o 

Oh 


o 

CO 

o 


GO 

a-) 

> 

o 

Oh 

GO 

0 

•  i— H 

o 


o 

+-» 

GO 

o 

GO 

•  i— i 

> 

o 

Oh 

GO 


O 


GO 
+-> 

GO 


a 


o 

-J— » 

GO  GO 
GO  Q 

is  <d 

Ctf  GO 


43 

o 

GO 


o 

o 


GO 

go  <u 


o 


O  —H 

zr  ° 

^  o 

in  to 

(U  GO 

43  to 

13  "5b 

>-<  o 

.2  o 

<D  GO 


o3 


> 

43 

<u 
4T> 

J-h 

4> 

=5 
O 

o 
c 

0) 
cS 

o 
4a 

a 

C 

c 


o 
o 

<U 
v- 

4^ 


o 
o 


ON 


go 
CD 

> 

o 

q.  d 
S  d 

^3  <D 

<D 
GO 


CD 


GO 

O 

CD 
X)  g 

Oh  C 

o 

ctf  O 

u 


<D 

•  i-H 

> 
O 

t-H 

Oh 
S  CD 


2 


CD  ^ 


> 
O 

GO 
GO 

<D 

a 

<D 
CD 

— H 


GO 
GO 
<D 


CD 


5-H 

o 


c3 


o 


o 

o  - 3 

>  §  g 


C/3 


3 


O 

"8 

*  T-H 

G 
O 


03 

o 


> 

O 

Mh  CD 


czs 


C3 

a 

CD 

5h 


3 


> 
O 


CD 

S 
g 

CD 

•  i-H 

g 

"o 

CD 

S 
a 

Oh 

o 


CO 
5-h 

CD 


c3 
G 

> 

co 


CD 

» 

CD 

CO 

CD 

a 
o 

G 
O 


CO 

5-H 

CD 


G 

CO 


GO 


Q 


CO 

CD 
G 
O 

G 
.G 
O 


G 

rG 
O 


o3 


> 

a> 

43 

o 

C 

o 
o 
c 

CD 

33 
Q 
43 

C 
03 
C 

c 


o 
o 

<U 
— 
03 
43 

CO 


CO 
CD 
G 
O 


o 
o 


( 


oo 


o 

o 

o 


a 

•  i-H 

g 

^  3 

s 

r— (  •  i-H 

S3  ^ 

3  o 

„  o 

Q  1 1 

U  o 


CO 

•  rH 


O 


6X) 


G  _ 

C  43 
o 


cd 


cd 


GO 

o 

GO 

HH 

Oh 

O 
43 


o 

•  1—1 

cd 

o 


CO  rH 


•  i-H 

o 

CO 
Oh 

o 

cd 

•  i—H 

o 


cd 

•  i— H 

CD 


o 

-+- > 
1-H 

o 


S-H 
•  i-H 

o 

CO 

(D 

Oh 

CO 

<D 
O 


C5 
s- 

O 
•  i—i 

> 

<U 
-O 

S-h 

§ 
o 

c 

<u 


g 

c 


o 


o 
o 


ON 


c 


d 

o 


o 


d 
o 


o 
o 


O 


c 

go 
CD 

GO 

O 
+-> 

O 

GO 

•  i-H 

GO 

> 

O 
O 


03 

•3  2 

CD  ^ 


o 


C3 


O 

CO 

O 


c  S 

o  o 

GO  h 

GO  TJ 

•r<  a-) 


ON 


GO 


GO 
+-> 

GO 

a3 


G  GO 
«  Oh 


O 
42 


O 

-4— > 

o 
c 

o 
43 

u 

X 
PQ 

re5 

CO 


o 


in 
43 


» 

-a 
o 


as 

J-h 

o 
■> 

re! 
X! 

<D 
X) 

<u 

-4— ' 

c 

O 
O 

e 

o 

<D 

c 

res 

a 
c 


o 
o 

1) 

Si 

JO 


o 
o 


On 


Cxi 


B 

> 
43 


2 
o 

CO 

>  CO 

2  o 

s  ^ 

^  s 

o  o 
a 

t3  .2 
o 

5-H 


.£3  (U 

is  a 


co 

O  o3 


5-H 

o 
13 

co 

•  rH 

& 

O 

•  i-H 
+-> 

> 


O 
O 

CO 

a-) 

a 
o 

> 


> 

43 

o 
cd 

13 

o 

•  l-H 

O 

a 

o 
o 


t/3 

4— » 

o 

o 
a-) 

a 

> 
O 

5-H 

Oh 


*-h 

O 

> 
o 

5-H 


o 
43 


o 

O 

+- > 

o 

a 

O 

o 

Oh 


2 
43 


CO 
i— H 

CO 

"c3 

CO 
CO 

13 
c 

C3 


o 

> 


T3 

CD 
CD 


5-H 

O 

'> 

5-r 
<D 
-t— > 

c 
o 

O 

e 

CD 

o 
c 

C 

c 


o 
o 

u 


O 
O 


OS 


O 

a 
B 


o 

•  l—H 


o 

O 


co 
co 
<D 

o 
o 


CO 
GO 

o 

O 

a 

CD 


a 

■4— » 

c 
o 

o 

CO 

O 

o 


O 

o 


03 

& 

co 


3 


5-H 

Oh 

co 
co 


co  (D 
03 


s 


3 


CO 

co  <D 

GO  O 

8  1 

c3  £ 


a-) 

o 

S  » 

"3  GO 
O 

<3  S-i 

o  o 


c3 


2  -9 


CD 


2 

CO 
CO 

c3 

CO 

<D 

+- » 

c3 
O 

o 

> 


cr 

A 


O 

GO 


o 


cr 

V 


o 

o 

GO 
O 

o 


> 

<D 
> 

C 
=3 
O 
O 

a 

o 

§ 

C 


o 
o 

(D 


OX) 


o 
o 


On 


( 


CO 

B 

CD 


0> 

> 

^3 


GO 

bX)  O 


CO 

*s 


s 

O 

o 


^  H 


S-H 

o 

S3 

£ 


o 


c/> 
0> 


O 

c/3 


IS 

o 
o 

cd 
+- > 

o 

S3 

£ 

GO 

in 
-t— > 
S3 

|  s 

oa  CD 


in 


S3 

c3 


o 


172 


J3 

in 

a 

CD 
CD 

CO 

(D 
CD 

-a 

CD 

a 
o 

^— > 
O 


co 
CD 


O 

CD 
+-> 

o 

o 
a 

CO 

S3 
Ph 

d 

•  rH 

a 
o 

cd 
co 

CO 

CD 
O 

o 

CD 

CD 
O 

o 


c 
o 


o 

O 

CD 
CD 

% 

O 


Ph 

+— > 

CO 

cd 

CD 
+-> 

cd 


O 

o 
o 


o 

-4— » 


P3 
J-i 
O 

'I 

-o 

-I— > 

d 

3 

O 

o 
d 

3s 

O 

<u 
d 

C3 

d 
d 


o 
o 


o 
o 


ON 


V 


CD 

e 

S3 

CD 
CD 


CD 
> 

S3 
43 


73 


GO 

-a 

I 

a 

4^ 

o 
o 

O 


CD 

o 

•  1— I 

> 

43 

<D 
43 


o 


o 

CO 
•  i— I 

<-h  CD 

£  a 

^  CD 


O 

CO 


O 

o 


o 
O 


CD 

o 

CD 


o 

5-1 

o 

i—i 

c 
o 

o 


o 

•  f-H 


00 

H 

U 


<Z5 

CD 

> 

o 

f-H 

in 
<D 

4-* 

o 
o 
> 
7d 

c3 


O 
43 


G 
O 


m 


GO 

o 

CO 
GO 


5 


C/3 
3 


cd 

a 

o 


(73 


cd 
43 

t: 

cd 

M  O 
cd 


o 

U  Oh 

CD 


j~i 
O 

43 
43 

<u 
-t— > 
c 

§ 

o 
c 

a 

o 

c 
c 


o 
o 

T3 
43 

fee 


o 
o 


ON 


e 


^  o 


3 


o 


X3 

^  -8 


CD 


> 

o 


CD 

bD 

o 

GO 
GO 

-  8 

go  ^ 

go  ^ 

4^  o 

dn  g 

"B  43 

d  d3 

O  3 

0  w 

2  S 

1  I 


<D 

o 

go 
go 


go 

GO 


<3-> 


S  1? 

go  (D 

s  ^ 


o 

o 


o 


o 

•  i-H 

d 

CD 


CD 
42 


O 

d 


O 

o 

CD  S 


^  go 

GO  S 
2 


> 

a 

43 

<D 
43 
s- 
CD 
-i—' 

§ 

O 

o 
d 

CD 

O 
43 

d 

d 
d 


CD 

o 

XI 
CD 


go  a> 


GO  _ 


GO  +-> 
5-H  IT} 

2  o 


OJO 


GO 

CD 


Oh 
I 

d 
o 
d 


d 
d 

X3 
43 


a  a 


o 
o 


On 


Lessons  Learned  When  Using 
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Jack  Wackwitz,  Ph.D. 


Jack  Wackwitz  received  his  Ph.D.  in  Psychology  from  the  University  of  Illinois  -  Champaign-  His 
academic  training  was  in  research  methodology  with  emphasis  in  measurement,  pyschometrics, 
statistics,  research  design,  and  mathematical  modeling,  in  the  contexts  of  social,  experimental,  and 
community  psychology.  He  has  spent  his  career  teaching  and  applying  his  methodological  expertise 
to  the  study  and  evaluation  of  human  social  problems  and  to  the  service  systems  designed  to  combat 
those  problems.  To  this  end,  he  has  been  involved  as  a  researcher  and  evaluator  in  the  areas  of 
education,  minority  issues,  substance  abuse,  public  health,  and  mental  health.  He  has  taught 
graduate  courses  in  methodology,  developed  innovative  programs  in  community  psychology,  helped 
to  design  and  evaluate  state  plans  for  substance  abuse  and  mental  health,  and  conducted  surveys  of 
disenfranchised  populations.  He  developed  a  comprehensive  evaluation  system  for  persons  with 
multiple  substance  abuse  problems,  including  clinical  charting,  client  assessment,  program 
assessment,  and  service  delivery.  He  is  currently  a  researcher  at  Colorado  Mental  Health  Services  - 
the  state  mental  health  authority  -  where  he  works  on  special  research,  evaluation  and  policy  projects 
including  two  CMHS  funded  projects.  The  16  State  Pilot  Project,  and  Cornerstone:  A  System  of 
Care  for  Families  and  Youth  with  Mental  Health  and  Juvenile  Justice  Needs.  Some  of  his  recent 
work  has  included:  Alternatives  to  hospitalization  for  persons  in  mental  health  crisis  in  rural  areas; 
High  risk  consumers  and  resource  reallocation;  Level-of-Care  and  Level-of-Impairment  Assessment; 
A  Multi-state  model  theoretic  analysis  of  the  MHSIP  consumer  survey;  and,  Colorado' s  Standardized 
Evaluation  System. 
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Joan  earned  a  bachelor's  degree  with  honors  in  Sociology  from  the  University  of 
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Patrick  Roohan  is  the  Director  of  the  Bureau  of  Quality  Management  and  Outcomes 
Research,  Office  of  Managed  Care,  New  York  State  Department  of  Health  (DOH).  As 
director,  he  is  responsible  for  the  evaluation  of  the  managed  care  initiative  and 
development  of  data  systems  for  research  and  quality  of  care  studies.  Mr.  Roohan  has  led 
a  team  of  researchers  in  designing  the  new  Medicaid  Encounter  Data  System 
specifications,  which  has  been  reviewed  and  modified  by  managed  care  organizations. 
He  has  played  an  integral  role  in  automating  the  Quality  Assurance  Reporting 
Requirements  and  provides  technical  assistance  on  clinical  study  design,  reporting  and 
analysis.  He  has  supervised  the  development  of  the  provider  network  system,  a  quarterly 
reporting  system  for  managed  care  organizations  describing  in  detail  their  network  of 
providers.  Mr.  Roohan  is  the  former  Research  Director  of  the  Quality  Initiative,  and  is 
involved  in  many  quality  measurement  projects  throughout  the  DOH.  He  has  extensive 
experience  analyzing  data  from  various  DOH  data  systems.  Mr.  Roohan  received  his 
Masters  of  Science  in  Operations  Research  and  Statistics  from  Rensselaer  Polytechnic 
Institute. 


ADMINISTERING  SURVEYS 


I.  PURPOSE  OF  THE  PROTOCOL 

Surveys  of  Medicaid  beneficiaries  or  other  entities  are  becoming  an  increasingly  common 
method  of  health  care  quality  measurement.  Surveys  can  provide  information  that  is  not  generally 
available  through  clinical  or  administrative  records.  Surveys  also  can  capture  beneficiaries', 
families',  caregivers',  providers',  or  other  relevant  parties'  perceptions  of  access  to  care  and 
quality  of  health  services.  However,  in  order  for  a  survey  to  produce  valid  and  reliable 
information  (upon  which  to  make  valid  and  reliable  judgements  about  health  care  access,  quality, 
and/or  timeliness),  the  design  and  administration  of  a  survey  must  be  carried  out  in  accord  with 
certain  fundamental  principles  of  survey  design  and  administration.  Therefore,  this  protocol 
describes  the  activities  that  must  be  undertaken  in  administering  surveys  if  those  surveys  are  to 
be  included  as  a  component  of  the  external,  independent  quality  review  required  under  federal 
law  for  Medicaid  managed  care  organizations  (MCOs)  and  prepaid  health  plans  (PHPs). 

II.  ORIGIN  OF  THE  PROTOCOL 

This  protocol  is  based  on  generally  accepted  principles  of  survey  design  and  administration. 
These  principles  are  typically  discussed  in  textbooks  and  other  academic  and  research 
publications.  The  publications  (or  references)  that  were  used  in  the  development  of  this  protocol 
are  located  in  Appendix  A. 

III.  PROTOCOL  OVERVIEW 

This  protocol  applies  to  all  types  of  surveys;  for  example,  beneficiary  surveys  and  provider 
surveys.  It  does  not  recommend  one  particular  survey  instrument,  sampling  methodology  or 
approach  to  analyzing  and  reporting  study  results  because  differences  in  the  purposes  of  surveys 
preclude  any  one  questionnaire,  sampling  methodology  and  analysis  and  reporting  strategy  from 
meeting  all  needs.  In  a  number  of  instances,  however,  the  protocol  provides  specific  information 
about  the  Consumer  Assessment  of  Health  Plans  Study  (CAHPS)  surveys  and  reporting  formats, 
which  were  developed  by  the  Agency  for  Healthcare  Research  and  Quality  (AHRQ)  (formerly  the 
Agency  for  Health  Care  Policy  and  Research  -  AHCPR)  through  cooperative  agreements  with  the 
RAND  Corporation,  Harvard  University,  and  the  Research  Triangle  Institute.  CAHPS  surveys 
are  frequently  used  by  States  to  evaluate  Medicaid  beneficiaries'  experiences  with  managed  care. 

This  protocol  assumes  that  the  State  Medicaid  agency  has  made  certain  key  decisions  pertaining 
to:  1)  survey  goals  and  objectives  (specifically,  the  question(s)  the  survey  is  designed  to  answer); 
2)  the  intended  audience(s)  for  survey  findings;  and  3)  the  selection  of  a  survey  instrument.  The 
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State  may  make  these  decisions  independently  or  in  consultation  with  the  States'  External 
Quality  Review  Organization  (EQRO)1  at  the  discretion  of  the  State. 

The  protocol  specifies  eight  activities  that  must  be  undertaken  as  part  of  a  methodologically 
sound  survey: 


IV.      PROTOCOL  ACTIVITIES 

The  eight  activities  comprising  this  protocol  are  discussed  below.  EQROs  should  use  a  work 
sheet  such  as  that  found  in  Appendix  B  to  assist  in  and  document  the  implementation  of  these 
activities. 


The  EQRO  must  have  a  clearly  written  statement  of  the  State's  purpose(s)  and  objective(s)  for 
administering  the  survey.  These  decisions  will  shape  the  survey  methodology,  administration, 
and  analytical  approach.  The  EQRO  should  have  in  writing  the  answers  to  the  following 
questions:  "What  does  the  State  want  to  learn  by  administering  the  survey?"  and  "What  does  the 
State  plan  to  do  with  the  survey  results?"  Typically,  States  will  want  to  obtain  survey 
information  to  help  monitor  and  evaluate  the  quality  of  care  provided  to  Medicaid  beneficiaries, 
assist  Medicaid  beneficiaries  to  choose  between  MCOs,  and/or  promote  quality  improvement 
initiatives.  Although  a  State  may  have  multiple  purposes  for  a  given  survey,  for  purposes  of  this 
protocol,  one  of  the  stated  purposes  of  the  survey  must  be  to  assess  ~  either  alone  or  in 
combination  ~  the  access,  quality,  and/or  timeliness  of  care  received  by  Medicaid  beneficiaries 
enrolled  in  individual  MCOs/PHPs. 

The  purpose(s)  and  objective(s)  of  the  survey  will  also  determine  the  survey's  "unit  of  analysis.  " 
The  unit  of  analysis  refers  to  the  type  of  entity  about  which  the  State  wishes  to  obtain 
information.  The  unit  of  analysis  may  be,  for  example:  individual  MCOs/PHPs,  the  State's 


A  State  may  choose  an  organization  other  than  an  EQRO  as  defined  in  Federal  regulation  to  administer  a 
survey.  However,  for  convenience,  in  this  protocol  we  use  the  term  "external  quality  review  organization"  (EQRO) 
to  refer  to  any  organization  administering  a  survey. 


1. 

2. 
3. 
4. 
5. 
6. 
7. 
8. 


Identification  of  survey  purpose(s)  and  objective(s) 
Identification  of  intended  survey  audience(s) 
Selection  of  the  survey  instrument 
Development  of  the  sampling  plan 

Development  of  the  strategy  for  maximizing  the  response  rate 
Implementation  of  the  survey 

Preparation  and  analysis  of  the  data  obtained  from  the  survey 
Documentation  of  the  survey  process  and  results 


ACTIVITY  1: 


Identification  of  survey  purpose(s)  and  objective(s). 
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managed  care  initiative  as  a  whole,  or  a  particular  group  of  providers.  Thus,  the  unit  of  analysis 
also  greatly  influences  the  design,  conduct,  and  analysis  of  the  survey.  Although  the  State's 
survey  may  address  multiple  units  of  analysis,  at  least  one  of  the  units  of  analysis  must  be  the 
individual  MCOs/PHPs  participating  in  the  survey;  i.e.,  the  survey  must  be  designed  to  provide 
representative  and  generalizable  information  about  individual  MCOs/PHPs. 

Following  the  determination  of  the  survey's  purposes(s)  and  unit(s)  of  analysis,  more  detailed 
objectives,  derived  from  the  purpose,  should  be  stated  in  writing.  Objectives  should  be  concise, 
explicit  and  measurable.  For  example,  if  the  purpose  of  the  study  is  to  discover  how  satisfied 
Medicaid  managed  care  enrollees  are  with  their  health  care,  more  specific  objectives  might 
include: 

•  Determining  if  enrollees  of  individual  MCOs/PHPs  are  satisfied  with  their  access  to 
speciality  care. 

•  Determining  if  enrollees  of  individual  MCOs/PHPs  are  involved  in  planning  for  their 
own  treatment. 

•  Determining  if  enrollees  of  individual  MCOs/PHPs  are  satisfied  with  the  quality  of  their 
interactions  with  their  primary  care  provider. 

ACTIVITY  2:  Identification  of  intended  survey  audience(s). 

The  EQRO  also  must  know  who  is/are  the  State's  intended  audience(s)  for  the  survey  results. 
The  intended  audience(s)  will  have  implications  for  the  design,  administration  and  analysis  of  the 
survey.  While  the  primary  audience  is  the  State  Medicaid  agency,  which  will  use  the  survey 
results  as  part  of  external  quality  review,  other  audiences,  identified  by  the  State,  could  include: 

•  Medicaid  beneficiaries  and  their  families  who  are  choosing  between  fee-for-service 
(FFS)  and  MCOs/PHPs,  or  among  Medicaid  MCOs/PHPs.  Survey  information 
increasingly  is  viewed  as  one  source  of  information  to  assist  the  consumer  in  making  an 
informed  choice.  This  requires  that  the  survey  be  designed  to  enable  MCO/PHP  -  to  - 
MCO/PHP  comparisons. 

•  MCO/PHP  managers  and  providers,  who  can  use  the  survey  results  to  identify  areas  of 
superior  health  service  delivery  as  well  as  areas  that  need  improvement.  When  surveys 
will  be  used  to  provide  information  to  MCOs/PHPs,  the  State  will  need  to  develop  a 
policy  that  specifies  whether  or  not  the  MCO/PHP  will  be  able  to  receive,  or  have  access 
to,  individual  enrollee  survey  results  or  just  summary,  plan-level  information  on  their 
enrollee's  survey  results  in  the  aggregate.  Providing  MCOs/PHPs  with  survey  results  for 
individual  enrollees  could  violate  enrollee  confidentiality  and  potentially  subject  enrollees 
for  adverse  action  by  the  MCO/PHP.  Even  if  the  enrollee's  name  or  other  identification  is 
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removed  from  the  survey  response  form,  there  remains  a  concern  that  a  MCO  might  still 
be  able  to  identify  a  particular  survey  respondent  by  his  or  her  responses.  Because  of  this, 
the  Medicare  program  does  not  allow  its  contracting  MCOs  to  receive  individual-level 
survey  data,  even  when  the  surveys  are  stripped  of  enrollee  identifiers.  Medicare  is, 
however,  establishing  a  mechanism  for  MCOs  to  request  special  analyses  of  their  own 
data  since  they  cannot  receive  the  individual-level  data  themselves. 

•         State  policy  makers  who  can  use  survey  findings  to  monitor  how  Medicaid  beneficiaries 
perceive  the  care  they  receive  under  the  State 's  managed  care  initiative.  In  this  case,  the 
survey  analysis  would  need  to  provide  information  on  the  State's  managed  care  initiative 
as  a  whole,  as  well  as  on  individual  MCOs/PHPs. 

It  is  important  for  the  State  to  clearly  identify  the  intended  audiences  to  the  EQRO,  as  this  will 
significantly  affect  the  format  of  the  report(s)  prepared  by  the  EQRO  for  the  State.  Because  of 
this,  the  State  will  need  to  identify  the  audiences  for  which  it  wants  the  EQRO  to  prepare  reports. 
Sometimes  the  State  will  want  the  EQRO  to  prepare  reports  only  for  the  State  Medicaid  agency, 
after  which  the  Medicaid  agency  will  prepare  reports  for  other  audiences.  In  other  situations,  the 
State  may  want  the  EQRO  to  prepare  survey  reports  in  multiple  formats  for  different  audiences. 

ACTIVITY  3:  Selection  of  the  survey  instrument. 

The  State  will  determine  the  survey  instrument  to  be  used.  The  State  may  do  so  independently  or 
in  consultation  with  the  EQRO,  at  the  discretion  of  the  State.  There  are  three  approaches  to 
selecting  a  survey  instrument  for  use:  1 )  use  an  existing  instrument;  2)  develop  a  new  instrument; 
and  3)  adapt  an  existing  instrument.  The  State's  decision  with  respect  to  these  choices  should  be 
consistent  with  the  survey  purposes,  objectives,  and  units  of  analysis,  and  should  promote  the 
collection  of  reliable  and  valid  data  (reliability  and  validity). 

Reliability  refers  to:  1)  the  internal  consistency  of  a  survey  instrument;  and  2)  the  reproducibility 
of  survey  results  when  the  survey  is  administered  under  different  conditions;  e.g.,  by  different 
people,  or  at  different  times.  Internal  consistency  requires  that  positive  statistical  correlations 
exist  between  questions  within  a  survey.  In  other  words,  a  respondent's  answers  to  particular 
survey  questions  should  not  contradict  the  respondent's  answer  to  other  survey  questions. 
Answers  should  be  consistent  across  related  questions.  Reproducibility  means  that  a  tool  yields 
the  same  results  in  repeated  trials.  For  example,  a  grocery  store  scale  should  show  the  same 
weight  each  time  the  same  bag  of  produce  is  measured.  If  it  shows  a  different  weight,  it  is 
unreliable.  Similarly,  a  survey  instrument's  questions  should  not  be  answered  differently 
depending  upon  who  is  administering  the  survey  or  in  what  location  or  manner  the  survey  is 
administered.  Survey  questions  also  should  be  clear  so  that  individuals  responding  to  the  same 
question  in  the  survey  will  not  interpret  the  same  question  differently. 

Validity  refers  to  the  degree  to  which  a  tool  captures  what  it  is  intended  to  measure.  Different 
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types  of  validity  measurement  include:  face  validity,  content  validity,  construct  validity,  and 
criterion-related  validity.  Face  validity  is  also  known  as  logical  validity  and  ensures  that  the  tool 
generates  common  agreement  and/or  mental  images  associated  with  a  concept  about  which  the 
survey  is  asking.  This  type  of  validity  does  not  involve  statistical  analysis,  but  is  an  opportunity 
to  determine  if  a  concept  is  well  addressed  by  a  question  or  questions  in  a  survey.  This  can  be 
accomplished  through  a  review  that  may  include  one  or  more  focus  groups  whose  primary 
purpose  is  to  agree/disagree  on  whether  the  survey,  on  its  face,  is  capturing  the  concept  to  be 
addressed  by  the  survey.  Content  validity  builds  on  face  validity  by  asking  whether  the  survey 
fully  captures  and  represents  the  concept  under  study.  This  type  of  validity  is  typically 
established  by  a  review  for  relevance  by  individuals  with  expertise  in  the  subject  matter 
addressed  in  the  survey.  Content  validity  also  is  not  established  using  statistical  procedures. 
Rather,  it  is  established  by  agreement  from  experts  in  the  topic  under  study. 

Construct  and  criterion  validity  are  more  sophisticated  measures  of  validity  and  require  more 
time  and  resources  to  assess.  Construct  validity  assesses  how  well  the  test  measures  the 
underlying  issue  (e.g.,  access  to  care,  satisfaction  with  care,  satisfaction  with  providers)  it  is 
supposed  to  measure.  Criterion  validity  measures  how  well  the  performance  on  a  test  reflects  the 
performance  on  what  the  test  is  supposed  to  measure.  There  are  two  types  of  criterion  validity: 
concurrent  and  predictive.  Concurrent  validity  uses  a  correlation  coefficient  to  compare  the 
survey  to  a  "gold  standard"  survey  for  measuring  the  same  variable.  Predictive  validity  measures 
the  survey's  ability  to  "predict"  future  outcomes  and  also  uses  a  correlation  coefficient  for 
comparison.  While  no  instrument  is  ever  determined  to  be  "perfect'  with  respect  to  reliability 
and  validity,  the  greater  amount  of  reliability  and  validity  testing,  and  the  greater  the  positive 
findings  with  respect  to  reliability  and  validity  that  are  associated  with  a  given  survey  instrument, 
the  greater  confidence  an  EQRO  can  have  in  a  survey's  findings. 

The  CAHPS  survey  instruments  and  reporting  formats  have  undergone  rigorous  testing  for 
reliability  and  validity,  including  focus  group  interviewing,  cognitive  interviewing,  and  field- 
testing.  However,  if  a  State  chooses  to  use  another  instrument,  develop  one  of  its  own,  or  modify 
an  existing  instrument,  at  a  minimum,  it  should  promote  reliability  and  validity  by  first 
establishing  face  and  content  validity,  then  pretesting  the  tool  for  reliability.  Face  and  content 
validity  can  be  established  by  convening  one  or  more  focus  groups  that  include  beneficiaries  and 
individuals  with  subject  matter  expertise.  Reliability  can  be  assessed  using  the  test-retest  method 
in  which  the  survey  is  administered  to  the  same  group  at  two  different  times.  A  correlation 
coefficient  is  calculated  and  indicates  the  reproducibility  of  results.  Correlation  coefficients  with 
r  values  at  or  above  0.70  indicate  good  reliability. 

Option:  using  existing  survey  instruments.  Because  of  the  resources  required  to  develop  and 
test  new  survey  tools,  many  States  will  choose  to  use  an  existing  survey  instrument  -  especially 
one  that  has  undergone  strong  reliability  and  validity  testing  such  as  CAHPS.  CAHPS  testing 
included  cognitive  testing  during  the  development  and  evaluation  phases,  calculation  of 
reliability  estimates  in  a  sample  of  Medicaid  enrollees  and  private  health  insurance  purchasers, 
and  convening  of  focus  groups  to  test  relevance  of  survey  concepts  and  items.  In  addition, 
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CAHPS  surveys  include  a  free,  easy-to-use  CAHPS  Survey  and  Reporting  Kit.  The  kit  contains 
a  set  of  mail  and  telephone  survey  questionnaires  including  Spanish  language  versions,  sample 
reporting  formats,  and  a  handbook  with  step-by-step  instructions  for  sampling,  administration, 
analysis,  and  reporting.  The  handbook's  instructions  are  easy  to  understand  and  allow  for 
flexibility  within  the  sample  and  analysis  design.  The  handbook  also  includes  a  comprehensive, 
statistical  software  package  as  well  as  the  telephone  number  and  e-mail  address  of  the  technical 
assistance  hotline. 

There  are  a  wide  variety  of  other  existing  questionnaires  that  can  be  selected  as  the  survey 
instrument.  While  some  have  undergone  extensive  reliability  and  validity  testing,  others  have 
been  developed  with  little  or  no  testing  for  reliability  and  validity.  Unless  the  developers  of  an 
existing  survey  can  describe  an  instrument's  reliability  and  validity  testing  and  results,  an 
instrument's  reliability  and  validity  should  always  be  suspect.  Even  pre-existing,  validated  survey 
instruments  may  not  have  been  validated  in  a  Medicaid  population.  Selection  of  instruments  not 
validated  in  a  Medicaid  population  may  not  yield  valid  or  reliable  results  for  this  population. 

Another  advantage  of  selecting  an  existing  instrument  is  that  use  of  the  same  questionnaire, 
methods,  and  analysis  for  surveys  of  MCOs/PHPs,  population  groups,  and  States,  permits  a  State 
to  compare  its  findings  against  those  of  other  studies.  For  example,  standardization  of  the  survey 
instruments,  methodology,  and  report  formats  in  CAHPS  provides  reliable  and  valid  measures 
that  can  be  compared  across  MCO/PHPs,  across  MCO/PHPs  and  fee-for-service  (FFS),  as  well 
as  across  a  wide  range  of  respondents  and  States. 

When  using  an  existing  survey  instrument  for  External  Quality  Review,  the  EQRO  should  be 
able  to  document  the  extent  of  reliability  and  validity  testing  of  the  survey  instrument. 

Option:  developing  new  survey  instruments.  In  some  cases,  a  State  may  decide  to  use  a  new 
survey  that  it  or  a  State  contractor  develops.  Usually  a  State  will  choose  to  develop  a  new  survey 
instrument  when  the  purposes  and  objectives  of  the  study  require  answers  to  questions  that  are 
not  addressed  by  existing  instruments.  A  well-designed,  State-specific  instrument  can  capture 
information  that  is  of  great  interest  and  relevance  to  the  questions  under  study.  However, 
assuring  the  reliability  and  validity  of  new  surveys  is  costly  and  takes  time.  Without  such 
reliability  and  validity  testing,  surveys  can  have  serious  methodological  shortcomings  and  be 
vulnerable  to  criticism.  A  1997  report  by  the  Department  of  Health  and  Human  Services  (DHHS) 
Inspector  General,  "Medicaid  Managed  Care:  The  Use  of  Surveys  as  a  Beneficiary  Protection 
Tool"  found  that  some  State-specific  surveys  were  designed  to  meet  too  many  objectives  and 
provided  little  useful  data.  When  developing  new  survey  instruments  and  methods  for  purposes 
of  External  Quality  Review,  the  State  Medicaid  agency  should  involve  an  individual  with 
experience  in  survey  design  and  methodology  and  assure  testing  of  the  instrument  for  validity 
and  reliability. 

Option:  adapting  existing  surveys.  States  also  may  decide  to  adapt  an  existing  questionnaire 
by  adding  or  deleting  items,  modifying  questions,  or  using  only  certain  groups  of  questions  to 
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respond  to  a  specific  issue  under  study.  This  approach  provides  States  with  the  flexibility  to  add 
or  change  the  data  to  be  collected,  while  providing  many  of  the  advantages  of  using  a  pre- 
existing questionnaire.  However,  this  approach  also  can  raise  questions  about  the  validity  and 
reliability  of  the  new  or  modified  questions,  as  well  as  the  survey  overall.  Validated  instruments 
are  tested  "as  a  whole,"  and  modifications  can  change  the  focus  and  purpose  of  the  tool. 
Adapting  an  existing  survey  is  easier  if  CAHPS  is  used,  as  the  CAHPS  survey  instruments  have 
been  specifically  designed  with  opportunities  to  customize  the  questionnaire  by  adding  questions 
from  a  list  of  supplementary  optional  items,  as  well  as  State-designed  questions.  Most  pre- 
existing questionnaires  are  not  designed  to  accommodate  such  modifications  and  require 
additional  testing.  Modifications  and  additional  testing  is  time  intensive  and  will  likely  result  in 
increased  costs.  When  a  State  undertakes  to  adapt  an  existing  tool  for  its  purposes,  it  should 
obtain  the  advice  of  an  individual  knowledgeable  in  survey  design  (preferably  knowledgeable 
about  the  original  survey)  to  advise  on  the  modification  and  how  to  appropriately  test  for 
reliability  and  validity. 

ACTIVITY  4:  Development  of  the  sampling  plan. 

When  the  State  decides  to  collect  information  directly  from  beneficiaries,  providers,  or  other  unit 
of  analysis,  it  must  decide  how  may  "units  "  will  need  to  be  surveyed.  A  State  could  decide  to 
take  a  "census; "  i.e.,  ask  every  individual  entity  which  the  survey  is  designed  to  represent,  to 
answer  the  survey  questions.  For  example,  a  State  could  decide  to  survey  all  Medicaid 
beneficiaries  or  all  Medicaid  participating  providers.  However,  because  the  cost  of  contacting  all 
these  entities  is  expensive,  surveys  typically  are  administered  to  only  a  portion  (a  "sample")  of 
the  target  population.  Whenever  a  sample,  as  opposed  to  a  census,  is  used,  it  is  very  important  to 
select  the  sample  members  in  a  way  that  represents  as  closely  as  possible  the  population  about 
whom  information  is  desired  -  both  those  included  in  the  sample,  and  those  not  selected  to 
respond  to  the  survey. 

Step  1 :  Identify  the  study  population. 

Sample  selection  begins  with  the  identification  of  the  population  to  be  studied;  e.g.,  all  Medicaid 
beneficiaries  enrolled  in  a  State's  Medicaid  managed  care  initiative,  or  all  children  with  special 
health  care  needs.  The  "population  "  represents  all  units  of  analysis  (sometimes  referred  to  as 
"the  universe  ").  A  good  sample  is  a  smaller  version  of  the  population  to  be  studied  and  is 
representative  of  the  population  in  terms  of  certain  pre-identified  characteristics  (e.g.,  race,  age, 
MCO/PHP  affiliation.) 

Step  2:  Define  the  sample  frame. 

Once  the  target  population  for  the  survey  is  identified,  the  population  from  which  the  sample  will 
be  drawn  must  be  further  refined.  This  is  the  "sample  frame.  "  The  sample  frame  lists  all 
members  of  the  study  population  eligible  for  the  study  and  is  used  to  select  the  sample.  Questions 
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used  to  define  individuals  eligible  for  the  survey  (the  sample  frame)  might  include,  for  example: 

Is  a  minimum  period  of  enrollment  in  a  MCO/PHP  necessary  for  individuals  who  will  be 
surveyed?  Must  the  beneficiary  be  continuously  enrolled  in  a  specific  MCO/PHP  for  a 
specified  period  of  time  to  be  considered  an  enrollee  for  purposes  of  the  survey? 
Frequently,  a  minimum  period  of  continuous  enrollment,  such  as  six  months,  is  required 
to  increase  the  likelihood  that  enrollees  have  had  contact  with  their  specific  MCO/PHP. 

Should  any  MCO  enrollees  be  excluded?  Why?  For  example,  if  a  study  population  is  to 
address  children  with  special  health  care  needs,  should  the  survey  be  limited  to  only 
children  receiving  SSI  or  Title  V,  or  should  it  address  all  children  that  a  MCO/PHP  might 
identify  as  having  special  health  care  needs,  or  should  it  include  all  children  whose  family 
might  identify  them  as  having  a  special  health  care  need? 

Should  only  one  member  per  household  be  surveyed? 

Will  children  be  surveyed? 

When  is  open  enrollment  and  how  will  it  impact  survey  administration? 
Step  3:  Determine  the  type  of  sampling  to  be  used. 

There  are  two  basic  types  of  statistical  sampling:  probability  sampling  and  nonprobability 
sampling.  Probability  sampling  leaves  selection  of  the  units  to  be  included  in  the  survey  up  to 
chance.  It  is  designed  to  prevent  any  biases  in  the  sample  selected  by  ensuring  that  each  segment 
of  the  study  population  is  represented  in  the  sample  to  the  extent  it  composes  part  of  the 
population  under  study.  Simple  random  sampling  is  the  most  common  form  of  probability 
sampling  used  in  survey  research.  Population  members  are  generally  assigned  a  number,  and 
random  numbers  generated  by  a  computer  or  table  of  random  numbers  are  used  to  select 
members  from  the  population.  This  sampling  approach  ensures  that  all  members  of  the  target 
population  have  an  equal  chance  of  selection.  The  sample  is  thus  assumed  to  be  fully 
representative  of  the  population. 

Stratified  random  sampling  also  may  be  used.  This  technique  calls  for  dividing  the  population 
(on  the  basis  of  prior  knowledge  about  the  population)  into  specific,  pre-  identified  strata  or 
subgroups  that  are  homogeneous  within  themselves  with  respect  to  certain  characteristics  (e.g., 
ethnicity,  age,  diagnoses).  A  random  sample  is  then  taken  from  each  stratum  or  subgroup. 
Stratification  is  done  both  to  improve  the  accuracy  of  estimating  the  total  population's 
characteristics  and  to  provide  information  about  the  characteristics  of  interest  within  subgroups. 
However,  stratified  random  sampling  requires  more  information  about  the  population,  and  can 
also  require  a  larger  total  sample  size.  As  a  result,  it  is  typically  more  expensive  than  simple 
random  sampling.  Stratified  sampling  may  also  involve  "weighting"  the  sample.  In  this  process,  a 
survey  would  select  a  disproportionately  larger  number  of  units  of  analysis  from  one  or  more  of 
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the  strata  to  allow  the  survey  to  produce  information  on  that  particular  strata;  e.g.,  individuals 
dually  receiving  both  Medicare  and  Medicaid. 

Non-probability  sampling  methods  are  based  on  the  decisions  of  those  administering  the  survey, 
and  not  on  random  chance.  For  example,  a  State  might  wish  to  obtain  information  about  new 
Medicaid  beneficiaries  by  surveying  all  new  beneficiaries  eligible  in  a  particular  month.  It  might 
wish  to  obtain  information  on  average  stage  of  pregnancy  at  time  of  the  establishment  of 
Medicaid  eligibility  by  examining  the  records  of  women  giving  birth  at  a  particular  hospital.  In 
such  cases,  the  information  obtained  by  the  State  will  have  been  biased.  Non-random  sampling 
methods  such  as  these  also  do  not  lend  themselves  to  statistical  analysis.  Because  of  the  risk  of 
biased  results  and  the  obstacles  to  statistical  analysis,  nonprobability  sampling  is  discouraged. 
However,  at  times  it  can  be  an  appropriate  and  efficient  way  of  collecting  needed  information. 

Step  4:  Determine  needed  sample  size. 

A  sample  only  collects  information  from  a  fraction  of  the  population.  Based  on  data  obtained 
from  the  sample,  estimates  are  produced  about  the  utilization  of  health  care  services,  satisfaction 
with  care,  quality  of  care  or  other  issues.  However,  these  estimates  are  at  risk  of  being 
"inaccurate"  to  the  extent  that:  (1)  they  are  different  from  the  population's  true  values  because 
only  a  portion  of  the  entire  population  was  involved  in  producing  the  estimates,  and  (2)  a 
different  sample  will  give  different  estimates.  Fortunately,  the  risk  of  inaccuracy  can  be  reduced 
by  carefully  selecting  the  sample  size. 

Two  factors  influence  the  determination  of  the  appropriate  sample  size  for  the  survey2:  (1)  the 
level  of  certainty  desired  for  the  estimate  to  be  produced  and  2)  the  margin  of  error  that  is  defined 
as  acceptable.  Combining  these  two  factors  establishes  the  parameters  for  the  estimates  to  be 
produced  by  the  survey  (e.g.,  a  survey  would  target  the  answers  to  its  questions  to  be  95% 
accurate  with  a  5%  margin  of  error).  The  greater  the  certainty  and  the  smaller  the  margin  of  error 
desired,  the  larger  the  sample  will  need  to  be  (and  the  more  expensive  the  survey  will  be).  In 
addition,  oversampling  (sampling  for  a  greater  number  of  entities  than  are  actually  needed  to 
produce  valid  estimates)  is  sometimes  undertaken,  to  compensate  for  an  anticipated  percent  of 
non-respondents  to  the  survey.  Because  of  these  and  other  factors;  e.g.,  whether  or  not  the  State 
intends  to  compare  MCOs/PHPs  to  one  another,  the  EQRO  will  need  to  determine  the  sample 
size  either  as  directed  by  the  State,  or,  if  the  State  allows  the  sample  size  to  be  determined  by  the 
EQRO,  in  consultation  with  a  qualified  statistician.  The  level  of  certainty  and  margin  of  error 
must  be  approved  by  the  State. 


2  A  third  factor  (the  true  frequency  of  the  occurrence  of  a  characteristic  within  a  population)  is  also  often 
used  to  calculate  the  sample  size  for  some  studies.  However,  since  a  survey  studies  multiple  questions,  and  the  true 
prevalence  of  responses  to  any  of  these  questions  may  be  variable  across  questions,  and  change  over  time,  the  most 
prudent  course  of  action  is  to  assume  the  need  for  the  maximum  sample  size  by  assuming  that  the  presence  or 
absence  of  the  characteristic  in  question  is  evenly  distributed  across  the  population. 
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If  the  CAHPS  survey  is  used,  the  EQRO  can  take  advantage  of  the  statistical  analysis  already 
performed  by  the  CAHPS  developers  and  follow  the  sampling  specifications  given  in  the  CAHPS 
handbook.  However,  these  will  differ  from  the  specifications  provided  by  the  National 
Committee  for  Quality  Assurance  (NCQA)  in  the  CAHPS  2.0H  manual.  (CAHPS  2.0H  is  an 
adaptation  of  CAHPS  developed  by  the  NCQA,  in  conjunction  with  AHCPR,  for  use  with 
NCQA's  Healthplan  Employer  Data  and  Information  Set  -  HEDIS).  The  CAHPS  2.0H 
questionnaire  is  identical  to  the  CAHPS  questionnaire  except  for  the  inclusion  of  the  HEDIS 
smoking  cessation  counseling  questions  as  mandatory  items.  This  difference  produces  different 
sampling  specifications  for  CAHPS  2.0  and  for  NCQA  CAHPS  2.0  H.  NCQA's  CAHPS  2.0H 
requires  a  larger  sample  size  in  order  to  obtain  adequate  numbers  of  respondents  who  can  answer 
the  smoking  questions. 

Step  5:  Specify  the  sample  selection  strategy. 

Once  the  sample  size  is  determined,  this  number  of  entities  must  be  selected  from  the  sample 
frame.  The  strategy  for  selecting  the  sample  should  specify  the  following: 

How  the  sample  will  be  drawn.  Once  the  type  of  sampling  has  been  determined,  the 
EQRO,  State,  or  some  other  entity  will  implement  the  sample  selection  procedures  to 
identify  the  specific  individuals  to  be  surveyed.  The  State  should  first  approve  sample 
selection  parameters  and  criteria.  If  the  EQRO  or  some  other  entity  is  drawing  the  sample, 
the  EQRO  should  review  the  specifications  prior  to  the  draw.  The  EQRO  will  review  the 
sample  selection  procedures  even  if  it  does  not  pull  the  sample.  This  should  include  a 
review  of  the  statistical  program  used  to  generate  the  sample,  a  scan  of  the  sample  frame 
files  for  face  validity,  and  the  generation  of  the  statistical  program  on  a  sample  from  the 
data  files. 

How  will  the  files/sample  be  delivered  and  in  what  format?  Information  technology  staff 
may  need  to  be  consulted  on  the  mechanism  to  be  used  to  deliver  the  sample  frame 
member  files  for  sample  selection  and  contact.  Typical  modes  of  delivery  include  file 
transfer  protocol  (FTP),  e-mail,  or  overnight  mail  on  disk  or  tape.  Formats  might  include 
CD-ROM,  Excel,  4  mm.dat,  or  reel  tape. 

Is  there  a  data  dictionary  for  the  sample  frame  or  sample  file?  A  data  dictionary  should 
be  provided  to  facilitate  immediate  use  of  the  sample  frame  or  sample  file.  The  data 
dictionary  is  a  listing  and  description  of  all  data  fields  found  in  an  electronic  or  hardcopy 
list  of  sample  frame  or  selected  sample  members. 

ACTIVITY  5:  Development  of  the  strategy  for  maximizing  the  response  rate. 
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The  EQRO  must  employ  a  strategy  for  maximizing  the  response  rate  that  addresses:  1 )  how 
respondents  in  the  sample  will  be  contacted;  and,  2)  how  the  response  rate  will  be  maximized 
and  calculated. 


Step  1:  Specify  the  strategy  for  contacting  individuals  to  be  surveyed. 

The  strategy  for  locating  and  contacting  individuals  who  are  to  be  surveyed  (target  respondents) 
should  be  developed  in  detail.  Surveys  can  be  administered  in  a  face-to-face  interview,  by 
telephone  interview,  or  by  a  mailed  questionnaire  to  be  completed  by  the  respondent  and  mailed 
back.  Because  of  the  importance  of  achieving  a  high  response  rate,  considerable  research  has 
been  conducted  on  these  approaches  to  survey  administration.  Recent  studies  indicate  the 
percentage  of  nonrespondents  is  increasing,  making  survey  research  more  difficult.  Although  in 
the  past  survey  researchers  believed  that  face-to-face  interviewing  produced  higher  response 
rates,  recent  research  challenges  this  view.  No  single  method  of  survey  administration  is 
believed  to  be  superior  in  all  circumstances.  Therefore,  the  type  of  survey  administration  chosen 
(e.g.,  face-to-face  interviewing,  telephone  or  mail)  should  reflect  careful  consideration  of  the 
response  rate  as  well  as  cost  and  time  issues.  Telephone  and  mail  surveys  usually  are  much  less 
expensive  than  in-person  interviewing.  In-person  interviews  are  expensive  because  of  the  time 
and  cost  required  when  using  experienced,  well-qualified  interviewers. 

Specific  data  needed  by  the  EQRO  to  administer  the  survey  should  be  identified;  e.g.,  an 
individual's  full  name,  address,  phone  number,  date  of  birth,  primary  language  and,  if  relevant, 
the  name  of  the  MCO/PHP  the  individual  is  enrolled  in,  and  the  length  of  time  of  enrollment.  It 
is  also  important  to  consider  the  completeness  of  the  information  needed  to  contact  individuals 
selected  for  the  survey,  and  to  what  extent,  if  any,  necessary  information  such  as  addresses  and 
telephone  numbers  can  be  verified  through  other  means;  e.g.,  the  MCO's/PHP's  enrollee  files. 
Plans  for  maximizing  the  probability  that  respondents  can  be  located  and  contacted  also  should 
be  outlined  (e.g.,  sending  names  in  the  sample  to  a  telephone  number  look-up  vendor  or  using  a 
change-of-address  data  base  vendor.) 

The  strategy  also  should  address  the  steps  to  be  undertaken  if  the  actual  rates  of  contacting 
individuals  are  lower  than  expected.  These  may  include  mailing  a  reminder  postcard  or  second 
survey,  or  a  follow-up  phone  call  to  individuals  failing  to  respond  to  a  mail  survey;  conducting 
repeat  calls  in  a  telephone  survey;  using  special  mail  delivery  services  for  respondents  who  have 
no  telephones;  or  tracking  and  following  up  on  the  number  of  "bad  number"  or  "undeliverable" 
cases  by  using  directory  assistance,  reverse  directories,  CD-ROM  address  directories,  national 
change-of-address  database  vendors,  and  telephone  number  look-up  vendors.  However,  the  data 
collection  strategy  will  also  need  to  consider  the  cost  implications  of  using  these  techniques.  For 
example,  the  CAHPS  developers  have  estimated  that  tracking  respondents  through  directory 
assistance  adds  $.75  per  case,  tracking  them  through  reverse  directories  adds  $3  per  case,  and 


HCFA/CMSO/QPMG/DQSM:  June  23,  2000  DRAFT 


I  I 


tracking  them  through  the  use  of  telephone  number  look-up  vendors  adds  $4  to  $8  dollars  per 
case. 

Step  2:  Maximizing  and  calculating  the  response  rate. 

The  response  rate  generally  is  the  number  of  entities  that  respond  to  the  survey  divided  by  the 
number  of  entities  selected  to  respond  to  the  survey.  However,  the  exact  definition  is  generally 
more  detailed  and  complex  that  this;  there  are  a  number  of  different  ways  to  calculate  response 
rates,  as  well  as  different  strategies  for  reporting  these  rates.  In  addition,  the  definition  of 
response  rate  may  differ  depending  on  mode  of  administration.  For  example,  mail  surveys 
typically  may  exclude  non-deliverable  addresses  from  the  denominator  while  telephone 
interviews  may  exclude  people  with  non-working  telephone  numbers  and  those  who  no  longer 
reside  in  the  household  from  the  denominator.  The  EQRO  and  the  State  must  determine  an 
appropriate  response  rate  methodology.  Given  the  response  rate  methodology  chosen,  the  State 
will  define  acceptable  levels  of  response  rates. 

The  response  rate  is  important  because  a  low  rate  reduces  the  ability  to  make  accurate 
conclusions  (statistical  inferences)  about  the  population  under  study.  Those  individuals  who  do 
not  respond  to  the  survey  may  differ  systematically  from  individuals  who  respond,  thereby 
biasing  the  results. 

Medicaid  beneficiaries  typically  have  a  low  survey  response  rate.  Medicaid  populations  tend  to 
have  lower  literacy  levels  than  average,  are  more  likely  to  speak  a  primary  language  other  than 
English,  are  highly  mobile,  and  have  high  rates  of  inaccurate  or  unavailable  telephone  numbers 
and  mailing  addresses.  As  a  result,  surveys  of  Medicaid  beneficiaries  tend  to  require  extra  effort 
to  maximize  response  rates.  These  efforts  can  include  using  an  expansive  contact  strategy  that 
emphasizes  tracking  methods,  and  a  mixed-mode  approach  to  survey  administration  that  typically 
combines  mail  and  telephone  survey  procedures.  However  even  with  these  additional  measures, 
response  rates  for  Medicaid  beneficiaries  are  likely  to  be  lower  than  for  many  other  populations. 
The  CAHPS  developers  suggest  that  the  target  response  rate  for  administering  CAHPS  to 
Medicaid  beneficiaries  should  range  from  40%  to  60%.  They  also  recommend  that  survey 
vendors  focus  on  strategies  that  promote  high  response  rates  and  develop  a  plan  of  correction 
actions  to  take  if  the  response  rate  falls  short  of  the  goal. 

No  matter  what  mode  of  survey  administration  is  selected,  research  indicates  that  a  number  of 
strategies  can  be  used  to  achieve  better  response  rates.  For  example,  if  mail  surveys  are  used,  all 
of  the  following  strategies  should  be  considered: 

•         Sending  an  advance  (pre-notification)  letter.  When  developing  an  in-house  letter,  the 
CAHPS  developers  suggest  that  a  number  of  points  should  be  stressed  in  the 
correspondence: 

Anonymity  and  confidentiality 
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How  the  enrollee  was  selected  for  participation 
How  the  results  will  benefit  the  respondent 
Why  the  survey  should  be  completed 
How  to  return  the  survey 

•  Using  follow-up  contacts  (e.g.,  reminder  postcards,  second  mailing  of  the  questionnaire, 
phone  contact  or  special  postage  mailing  of  a  second  questionnaire); 

•  Emphasizing  survey  sponsorship  (e.g.,  to  indicate  State  agency  sponsorship,  use  a  cover 
letter  on  State  government  letterhead  signed  by  the  Agency  Director; 

•  Using  personalized  correspondence  with  respondents  (e.g.,  addressing  all  correspondence 
to  the  respondent); 

•  Providing  stamped  return  envelopes;  and 

•  Using  special  postage  for  replacement  questionnaire  mailings  (e.g.,  overnight  mail  or 
certified  mail). 

Translation  of  surveys  and  correspondence  into  languages  other  than  English  also  may  be 
important  for  Medicaid  populations.  If  not  addressed,  language  issues  may  lead  to  lower 
response  rates  because  surveys  cannot  be  completed  if  they  cannot  be  read.  Failure  to  provide 
surveys  in  the  respondent's  primary  language  may  also  result  in  excluding  particularly  vulnerable 
segments  of  the  populations.  Unfortunately,  the  primary  language  of  specific  individuals  in  the 
survey  sample  may  be  difficult  to  determine.  One  solution  may  be  to  translate  a  sentence  in  the 
introductory  letter  and  provide  a  phone  number  for  the  respondent  to  call  for  more  information. 
During  the  call,  the  respondent  may  request  a  translated  survey,  complete  the  survey  over  the 
phone,  or  schedule  a  more  convenient  telephone  interview  time.  However,  survey  translations 
and  beneficiary  telephone  hot  lines  can  have  significant  cost  implications  for  the  project. 

The  survey  methodology  should  specify  the  required  response  rate  established  by  the  State, 
procedures  for  handling  missing  data,  and  the  specific  methods  for  calculating  both  raw  and 
adjusted  response  rates.  For  example: 

Raw  response  rates  can  be  calculated  by  : 

 Number  of  completed  surveys  

Total  number  of  targeted  survey  respondents 

Adjusted  response  rates  can  be  calculated  by: 

 Number  of  completed  surveys  
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Total  number  of  targeted  survey  respondents  minus  deceased  and  ineligible  due  to 
hospitalization,  cognitive  impairment,  or  other  reasons  specified  in  the  methodology 


ACTIVITY  6:  Implementation  of  the  survey. 

Based  on  the  information  obtained  in  Activities  1,  2,  and  3,  and  the  decisions  made  in  activities  4 
and  5,  the  EQRO  should  prepare  a  work  plan  to  govern  the  implementation  of  the  survey.  The 
work  plan  should  specify  routine  aspects  of  project  management  including  key  staff  and  their 
responsibilities,  timelines,  and  deliverables;  the  degree  to  which  each  phase  of  the  survey  process 
will  be  documented;  and  a  plan  of  action  for  identifying  and  resolving  problems.  It  should  also 
address  final  report  production  including  format,  content,  specification  of  any  needed  approval 
by  the  State  or  other  entities,  and  number  of  reports  to  be  submitted.  It  should  include  a 
description  of  any  reports  to  be  publicly  released  by  the  EQRO,  if  this  is  part  of  the  EQRO's 
scope  of  work. 

Key  methodological  issues  to  be  addressed  in  the  work  plan  include: 

•  procedures  to  be  followed  in  formatting,  reproducing,  and  distributing  the  survey 
questionnaire; 

•  procedures  for  assuring  the  confidentiality  of  the  data; 

•  data  collection,  data  entry,  and  internal  quality  control  methods;  The  EQRO  should 
determine  with  the  State  procedures  for  handling  responses  that  fail  edit  checks,  the 
treatment  of  missing  data,  and  the  handling  of  procedures  for  determination  of 
usable/complete  surveys. 

•  data  analysis  plan  including  statistical  methodology;  and, 

•  production  of  data  files  and  their  format  and  delivery,  if  applicable. 
The  work  plan  should  be  approved  by  the  State  prior  to  implementation. 

If  it  is  feasible,  the  survey  vendor  should  provide  the  State  with  a  mock-up  of  survey  results  prior 
to  administering  the  survey.  This  will  help  assure  that  the  information  to  be  produced  by  the 
survey  is  consistent  with  the  State's  planned  use  of  survey  results.  The  implementation  of  the 
survey  should  be  in  accord  with  the  State-approved  work  plan. 
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ACTIVITY  7:  Preparation  and  analysis  of  data  obtained  from  the  survey. 

Once  the  surveys  have  been  completed  and  returned  to  the  EQRO,  preparation  of  the  data  for 
analysis,  and  data  analysis  occurs.  This  includes  data  quality  control  procedures  (e.g.,  cleaning 
and  editing),  data  analysis,  and  production  of  data  files. 

Data  quality  control.  The  EQRO  should  develop,  implement,  and  document  quality  control 
procedures  for  all  survey  administration  activities.  In  particular,  quality  control  should  address: 
recording  receipt  of  mail  surveys;  telephone  administration  of  surveys;  handling  of  respondent 
information  and  assistance  telephone  calls;  coding,  editing,  and  keying  in  or  optical  scanning  of 
survey  data;  data  tabulation  and  analysis;  and  preparation  of  survey  reports. 

Data  analysis.  Each  MCO/PHP  should  be  treated  as  the  basic  unit  of  analysis  and  reporting.  The 
analysis  should  include  simple  statistical  procedures  such  as  calculation  of  measures  of  central 
tendency  and  frequency  distributions  for  specific  survey  questions.  In  addition,  differences  in 
survey  results  among  MCOs/PHPs  may  be  examined  using  standard  statistical  tests.  For 
example,  the  F-test  may  be  used  to  determine  if  the  average  scores  of  any  MCO/PHP  differ 
significantly  from  that  of  other  MCOs/PHPs.  The  F-test  tests  the  hypothesis  that  all  MCO/PHP 
means  (averages)  are  equal  by  comparing  the  relative  size  of  two  variances.  The  F-test  will  be 
significant  if  at  least  one  MCO  or  PHP  mean  is  statistically  different  from  any  other  MCO  or 
PHP  mean.  If  the  F-test  indicates  that  there  are  significant  differences,  a  series  of  t-tests  may  be 
conducted  to  identify  which  MCO/PHP  is  different  from  other  MCO/PHPs.  The  t-test  tests  the 
hypothesis  that  two  specific  MCO/PHP  means  are  significantly  different  from  each  other.  A 
significant  t-test  indicates  that  the  two  MCO/PHPs  represent  two  populations  with  different 
means. 

In  addition,  an  EQRO  may  want  to  consider  multivariate  regression  and/or  analysis  of  co variance 
techniques  to  further  identify  meaningful  differences  between  MCOs/PHPs  while  controlling  for 
respondent  characteristics.  For  example,  a  driver  analysis  and  a  quadrant  analysis  may  be  used  to 
gain  more  insight  into  the  current  survey  measures  and  possible  approaches  to  future 
improvement.  Driver  analysis  refers  to  statistical  techniques  that  may  be  used  to  identify  the 
important  factors  or  "drivers"  that  significantly  affect  some  outcome  measure.  In  this  context,, 
the  driver  analysis  will  identify  the  most  important  factors  (drivers)  that  significantly  affect 
overall  scores.  Individual  questions  can  be  analyzed  to  determine  their  effect  on  overall  scores. 
These  identified  drivers,  along  with  their  current  measures,  will  then  be  subjected  to  a  quadrant 
analysis  which  compares  the  importance  of  the  drivers  against  the  current  survey  measures  on 
these  drivers.  For  example,  if  access  to  specialists  has  been  identified  as  a  driver  of  patient 
satisfaction,  then  depending  on  the  importance  of  access  as  a  driver  and  the  MCO  current 
performance  on  access,  there  could  be  four  different  combinations  or  quadrants:  ( 1 )  high 
importance  and  high  performance;  (2)  high  importance  and  low  performance;  (3)  low  importance 
and  low  performance;  and  (4)  low  importance  and  high  performance.  High  importance  means 
that  health  care  access  has  an  important  effect  on  satisfaction,  and  high  performance  means  that 
health  care  access  has  been  well  addressed.  Therefore,  quadrant  ( 1 )  identifies  strength;  i.e., 
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satisfactory  performance  on  a  crucial  driver.  Similarly,  quadrant  (2)  identifies  areas  needing 
improvement;  quadrant  (3)  and  (4)  are  areas  of  low  leverage  and  insignificance.  The  most 
important  insight  that  can  be  gained  through  quadrant  analysis  is  the  identification  of 
improvement  opportunities.  MCOs/PHPs  can  then  make  strategic  decisions  about  where  to 
devote  their  efforts  and  resources  in  order  to  improve  MCO/PHP  scores. 

Consistent  with  the  purposes  and  objectives  of  the  survey,  populations  within  each  MCO/PHP 
could  also  be  analyzed  and  reported  separately.  For  instance,  the  State  may  be  interested  in 
whether  any  particular  variable  differs  significantly  across  geographic  locations,  demographic 
groups,  socio-economic  groups,  or  other  identifiable  subgroups  (e.g.,  frequent  and  infrequent 
users  of  health  care  services). 

Survey  results  may  be  adjusted  for  external  factors  that  influence  results.  For  example,  studies 
suggest  that  age,  education,  and  self-reported  health  status,  among  others,  have  a  significant 
effect  on  patient  satisfaction. 

Regardless  of  the  statistical  analyses  performed,  the  display  of  survey  findings  is  important. 
Statistical  graphs  should  accompany  narrative  text  to  aid  comparison  and  interpretation.  For 
example,  bar  graphs  and  comparison  charts,  such  as  those  recommended  by  CAHPS,  convey 
important  information  about  the  performance  of  each  MCO/PHP  and  indicate  meaningful 
differences  among  MCOs/PHPs. 

ACTIVITY  8:         Documentation  of  the  survey  process  and  results. 

The  EQRO  should  prepare  and  submit  reports  to  the  State  that  document  the  survey  process  and 
results,  including: 

1 .  survey  purpose  and  objectives; 

2.  technical  methods  of  survey  implementation  and  analysis  (Activities  2  -  7); 

3.  data  obtained; 

4.  conclusions  drawn  from  the  data; 

5.  a  detailed  assessment  of  each  MCO/PHP 's  strengths  and  weaknesses  with  respect 
to  (as  appropriate)  access,  quality,  and/or  timeliness  of  health  care  furnished  to 
Medicaid  enrollees; 

6.  as  the  State  determines  methodologically  appropriate,  comparative  information 
about  all  MCOs/PHPs. 

The  final  report  also  should  include  a  detailed  description  of  survey  methodology  problems, 
lessons  learned,  and  recommendations  for  improving  future  efforts.  If  the  scope  of  work  with  the 
State  requires  the  EQRO  to  produce  a  report  on  survey  results  for  public  reporting,  the  content 
and  format  of  this  report  should  conform  to  the  mock-up  specifications  previously  approved  by 
the  State.  Recently,  many  States  have  shown  increasing  interest  in  using  survey  data  for  public 
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reporting.  However,  fitting  survey  information  into  the  design  and  distribution  of  other 
beneficiary  materials  and  agreeing  on  the  report's  length,  design,  and  graphics  can  prove 
difficult.  Therefore,  it  is  important  for  the  EQRO  and  the  State  to  agree  on  how  to  present  the 
results  before  the  production  of  the  reports. 
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VALIDATING  SURVEYS 


I.       PURPOSE  OF  THE  PROTOCOL 

Surveys  of  Medicaid  beneficiaries  or  other  entities  are  becoming  an  increasingly  common 
method  of  health  care  quality  measurement.  Surveys  can  provide  information  that  is  not  generally 
available  through  clinical  or  administrative  records.  Surveys  also  can  capture  beneficiaries', 
families',  caregivers',  providers',  or  other  relevant  parties'  perceptions  of  access  to  care  and 
quality  of  health  services.  However,  in  order  for  a  survey  to  produce  valid  and  reliable 
information  (upon  which  to  make  valid  and  reliable  judgements  about  health  care  access,  quality, 
and/or  timeliness),  the  design  and  administration  of  a  survey  must  be  carried  out  in  accord  with 
certain  fundamental  principles  of  survey  design  and  administration. 

If  a  State  desires  its  External  Quality  Review  Organization  (EQRO)  to  incorporate  the  findings  of 
a  survey  (one  that  is  not  administered  by  the  EQRO  at  the  direction  of  the  state  Medicaid  agency) 
into  the  required  EQRO  activity  of  "analysis  and  evaluation  .  .  of  aggregated  information  on 
timeliness,  access,  and  quality  of  the  health  care  services  furnished  to  Medicaid  recipients  by 
each  MCO  .  .  ,"!  the  EQRO  must  determine  the  extent  to  which  the  survey  has  produced  valid 
and  reliable  information.  This  determination  is  necessary  in  order  for  the  EQRO  to  determine 
how  much  significance  to  place  on  a  particular  survey's  findings  when  the  EQRO  is  aggregating 
and  analyzing  information  from  multiple  sources  about  access,  quality  and  /  or  timeliness  of  care. 
This  protocol  describes  the  steps  that  need  to  be  undertaken  by  an  EQRO  to  evaluate  the 
methodologic  soundness  of  a  particular  survey,  if  the  survey  is  to  be  included  as  a  component  of 
the  external,  independent  quality  review  required  under  federal  law  for  Medicaid  managed  care 
organizations  (MCOs)  and  prepaid  health  plans  (PHPs).  In  HCFA's  proposed  rule  on  external 
quality  review  of  Medicaid  MCOs  and  PHPs,  administration  or  validation  of  provider  surveys  of 
quality  of  care  is  identified  as  an  optional  EQRO  activity,  to  be  used  at  the  discretion  of  the 
State.2 


II.        ORIGIN  OF  THE  PROTOCOL 

This  protocol  is  a  variation  of  the  protocol  on  "Administering  Surveys"  and  is  based  on  the  same 
generally  accepted  principles  of  survey  design  and  administration  that  guided  that  protocol.  This 
protocol  was  also  guided  by  the  same  academic  and  research  publications.  These  are  cited  in 
Appendix  A. 


Proposed  rule  on  Medicaid  Program;  External  Quality  Review  of  Medicaid  Managed  Care  Organizations. 
Federal  Register  Vol.64,  No  230  December  1,  1999. 

2Ibid. 
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III.       PROTOCOL  OVERVIEW 


This  protocol  applies  to  all  types  of  surveys;  for  example,  beneficiary  surveys  and  provider 
surveys.  It  does  not  assume  the  use  of  one  particular  survey  instrument,  sampling  methodology  or 
approach  to  analyzing  or  reporting  study  results  because  differences  in  the  purposes  of  surveys 
preclude  any  one  questionnaire,  sampling  methodology  and  analysis  and  reporting  strategy  from 
meeting  all  needs.  In  a  number  of  instances,  however,  the  protocol  addresses  the  Consumer 
Assessment  of  Health  Plans  Study  (CAHPS)  surveys  and  reporting  formats,  which  were 
developed  by  the  Agency  for  Healthcare  Research  and  Quality  (AHRQ)  (formerly  the  Agency  for 
Health  Care  Policy  and  Research  -  AHCPR)  through  cooperative  agreements  with  the  RAND 
Corporation,  Harvard  University,  and  the  Research  Triangle  Institute.  CAHPS  surveys  are 
frequently  used  by  States  to  evaluate  Medicaid  beneficiaries'  experiences  with  managed  care. 

In  this  protocol,  survey  validation  is  limited  to  review  of  survey  procedures.  It  does  not  include 
collecting  survey  data  anew  from  the  initial  survey  respondents  to  verify  their  responses.  Time, 
resource  requirements,  and  the  potential  demands  on  survey  respondents  typically  make  this 
activity  infeasible. 

The  protocol  specifies  seven  activities  that  must  be  undertaken  to  assess  the  methodologic 
soundness  of  a  given  survey: 

1 .  Review  survey  purpose(s)  and  objective(s) 

2.  Review  intended  survey  audience(s) 

3.  Assess  the  reliability  and  validity  of  the  survey  instrument 

4.  Assess  the  sampling  plan 

5.  Assess  the  adequacy  of  the  response  rate 

6.  Review  survey  data  analysis  and  findings/conclusions 

7.  Document  evaluation  of  survey 

IV.        PROTOCOL  ACTIVITIES 

The  eight  activities  comprising  this  protocol  are  discussed  below.  EQROs  should  use  a  work 
sheet  such  as  that  found  in  Appendix  B  to  assist  in  performing  and  documenting  the  validation 
process  for  the  survey  under  review. 

ACTIVITY  1:        Review  survey  purposes(s)  and  objective(s). 

In  order  to  have  a  sufficient  understanding  of  the  design,  administration,  and  analysis  of  the 
survey,  the  EQRO  should  communicate  with  the  entity(ies)  that  administered  the  survey  to 
review  and  understand  key  decisions  about  the  survey's  purpose(s)  and  objective(s).  These 
decisions  will  have  shaped  the  survey  methodology,  administration,  and  analytical  approach.  The 
EQRO  should  be  able  to  answer  the  following  question:  "What  did  the  survey  sponsor  want  to 
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learn  by  administering  the  survey?"  Although  a  survey  sponsor  may  have  multiple  purposes  for  a 
given  survey,  for  purposes  of  this  protocol,  one  of  the  stated  purposes  of  the  survey  must  have 
been  to  assess  ~  either  alone  or  in  combination  —  the  access,  quality,  and/or  timeliness  of  care 
received  by  Medicaid  beneficiaries  enrolled  in  individual  MCOs/PHPs. 

The  purpose(s)  and  objective(s)  of  the  survey  will  also  have  determined  the  survey's  "unit  of 
analysis.  "  The  unit  of  analysis  refers  to  the  type  of  entity  about  which  the  survey  sponsor  wishes 
to  obtain  information.  The  unit  of  analysis  maybe,  for  example:  individual  MCOs/PHPs,  the 
State's  managed  care  initiative  as  a  whole,  or  a  particular  grouping  of  providers.  Thus,  the  unit  of 
analysis  also  greatly  influences  the  design,  administration,  and  analysis  of  the  survey.  Although 
the  survey  under  review  may  have  addressed  multiple  units  of  analysis,  at  least  one  of  the  units  of 
analysis  must  have  been  individual  MCOs/PHPs  participating  in  the  survey;  i.e.,  the  survey  must 
have  been  designed  to  provide  representative  and  generalizable  information  about  individual 
MCOs/PHPs. 

Following  from  the  survey's  purposes(s)  and  unit(s)  of  analysis,  more  detailed  objectives  should 
have  been  explicitly  stated  in  writing.  Ideally,  objectives  should  be  concise,  explicit  and 
measurable.  For  example,  if  the  purpose  of  the  study  was  to  discover  how  satisfied  Medicaid 
managed  care  enrollees  were  with  their  health  care,  more  specific  objectives  might  include: 

•  Determining  if  enrollees  of  individual  MCOs/PHPs  are  satisfied  with  their  access  to  speciality 
care. 

•  Determining  if  enrollees  of  individual  MCOs/PHPs  are  involved  in  planning  for  their  own 
treatment. 

•  Determining  if  enrollees  of  individual  MCOs/PHPs  are  satisfied  with  the  quality  of  their 
interactions  with  their  primary  care  provider. 

The  survey  sponsor  should  provide  documentation  of  the  survey  purpose(s),  explicitly-stated 
objectives,  and  a  description  of  the  unit  of  analysis. 

ACTIVITY  2:        Review  intended  audience(s)  for  survey  findings. 

The  EQRO  also  should  have  an  understanding  of  the  intended  audiences  for  the  survey  results. 
Typically,  the  audiences  may  include: 

•  State  Medicaid  agencies  who  plan  to  use  the  survey  results  as  part  of  their  contracting  strategy 
for  improving  delivery  of  health  services; 

•  Medicaid  beneficiaries  and  advocates  who  are  choosing  between  FFS  and  MCOs/PHPs  or 
among  Medicaid  MCOs/PHPs.  Survey  information  increasingly  is  viewed  as  one  method  for 
allowing  the  consumer  to  make  an  informed  choice.  This  type  of  survey  effort  would  typically 
enable  comparisons  of  MCOs/PHPs; 
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•  MCO/PHP  managers  and  providers  who  can  use  the  survey  results  to  identify  areas  of 
superior  health  service  delivery  performance  as  well  as  areas  that  need  improvement. 

•  State  policy  makers  who  can  use  survey  findings  to  monitor  how  Medicaid  beneficiaries 
perceive  the  care  they  receive. 

The  intended  audience  for  a  survey  can  influence  the  selection  of  the  survey  instrument,  the 
wording  of  individual  survey  questions,  the  type  of  data  analysis  performed,  and  the  selection 
and  presentation  of  particular  survey  findings.  It  also  can  inadvertently  introduce  bias  in  the 
survey  process.  The  EQRO  should  understand  the  intended  use  of  the  survey,  as  one  part  of  its 
overall  evaluation  of  the  reliability  and  validity  of  the  survey. 

ACTIVITY  3:        Assess  the  reliability  and  validity  of  the  survey  instrument. 

A  survey  may  have  been  administered  using:  1 )  a  pre-existing  instrument  that  was  developed 
prior  to  the  survey  for  use  by  others;  2)  a  new  instrument  developed  specifically  for  the  particular 
survey  under  review;  or  3)  an  adaptation  of  a  pre-existing  instrument.  Regardless  of  which  of 
these  three  options  a  survey  sponsor  employed,  the  EQRO  will  need  to  determine  the  validity  and 
reliability  of  the  survey  instrument  as  a  critical  step  in  deciding  how  much  confidence  to  place  in 
the  survey's  findings. 

Reliability  refers  to:  1)  the  internal  consistency  of  a  survey  instrument;  and  2)  the  reproducibility 
of  survey  results  when  the  survey  is  administered  under  different  conditions;  e.g.,  by  different 
people,  or  at  different  times.  Internal  consistency  requires  that  positive  statistical  correlations 
exist  between  questions  within  a  survey.  In  other  words,  a  respondent's  answers  to  particular 
survey  questions  should  not  contradict  the  respondent's  answer  to  other  survey  questions. 
Answers  should  be  consistent  across  related  questions.  Reproducibility  means  that  a  tool  yields 
the  same  results  in  repeated  trials.  For  example,  a  grocery  store  scale  should  show  the  same 
weight  each  time  the  same  bag  of  produce  is  measured.  If  it  shows  a  different  weight,  it  is 
unreliable.  Similarly,  a  survey  instrument's  questions  should  not  be  answered  differently 
depending  upon  who  is  administering  the  survey  or  in  what  location  or  manner  the  survey  is 
administered.  Survey  questions  also  should  be  clear  so  that  individuals  responding  to  the  same 
question  in  the  survey  will  not  interpret  the  same  question  differently. 

Validity  refers  to  the  degree  to  which  a  tool  measures  what  it  is  intended  to  measure.  Different 
types  of  validity  measurements  include:  face  validity,  content  validity,  construct  validity,  and 
criterion-related  validity.  Face  validity  is  also  known  as  logical  validity  and  ensures  that  the  tool 
generates  common  agreement  and/or  mental  images  associated  with  a  concept  about  which  the 
survey  is  asking.  This  type  of  validity  does  not  involve  statistical  analysis,  but  is  an  opportunity 
to  determine  if  a  concept  is  well  assessed  by  a  question  or  questions  in  a  survey.  This  can  be 
accomplished  through  a  review  that  may  include  one  or  more  focus  groups  whose  primary 
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purpose  is  to  agree/disagree  on  whether  the  survey,  on  its  face,  is  capturing  the  concept  to  be 
addressed  by  the  survey.  Content  validity  builds  on  face  validity  by  asking  whether  the  survey 
fully  captures  and  represents  the  concept  under  study.  This  type  of  validity  is  typically 
established  by  a  review  for  relevance  by  individuals  with  expertise  in  the  subject  matter 
addressed  in  the  survey.  Content  validity  also  is  not  established  using  statistical  procedures. 
Rather,  it  is  established  by  agreement  from  experts  in  the  topic  under  study. 

Construct  and  criterion  validity  are  more  sophisticated  measures  of  validity  and  require  more 
time  and  resources  to  assess.  Construct  validity  assesses  how  well  the  test  measures  the 
underlying  issue  (e.g.,  access  to  care,  satisfaction  with  care,  satisfaction  with  providers)  it  is 
supposed  to  measure.  Criterion  validity  measures  how  well  the  performance  on  a  test  reflects  the 
performance  on  what  the  test  is  supposed  to  measure.  There  are  two  types  of  criterion  validity: 
concurrent  and  predictive.  Concurrent  validity  uses  a  correlation  coefficient  to  compare  the 
survey  to  a  "gold  standard"  survey  for  measuring  the  same  variable.  Predictive  validity  measures 
the  survey's  ability  to  "predict"  future  outcomes  and  also  uses  a  correlation  coefficient  for 
comparison.  While  no  instrument  is  ever  determined  to  be  "perfect'  with  respect  to  reliability  and 
validity,  the  greater  amount  of  reliability  and  validity  testing,  and  the  greater  the  positive  findings 
with  respect  to  reliability  and  validity  that  are  associated  with  a  given  survey  instrument,  the 
greater  confidence  an  EQRO  can  have  in  a  survey's  findings. 

The  CAHPS  survey  instruments  and  reporting  formats  have  undergone  rigorous  testing  for 
reliability  and  validity,  including  focus  group  interviewing,  cognitive  interviewing,  and  field- 
testing.  If  a  survey  sponsor  chose  to  use  another  instrument,  develop  one  of  its  own,  or  modify 
an  existing  instrument,  at  a  minimum,  it  should  promote  reliability  and  validity  by  first 
establishing  face  and  content  validity,  then  pretesting  the  tool  for  reliability.  Face  and  content 
validity  can  be  established  by  convening  one  or  more  focus  groups  that  include  beneficiaries  and 
individuals  with  subject  matter  expertise.  Reliability  can  be  assessed  using  the  test-retest  method 
in  which  the  survey  is  administered  to  the  same  group  at  two  different  times.  A  correlation 
coefficient  is  calculated  and  indicates  the  reproducibility  of  results.  Correlation  coefficients  with 
r  values  at  or  above  0.70  indicate  good  reliability. 

Existing  survey  instruments.  Because  of  the  resources  required  to  develop  and  test  new  survey 
tools,  many  survey  sponsors  will  choose  to  use  an  existing  survey  instrument  -  especially  one  that 
has  undergone  strong  reliability  and  validity  testing  such  as  CAHPS.  CAHPS  testing  included 
cognitive  testing  during  the  development  and  evaluation  phases,  calculation  of  reliability 
estimates  in  a  sample  of  Medicaid  enrollees  and  private  health  insurance  purchasers,  and 
convening  of  focus  groups  to  test  relevance  of  survey  concepts  and  items.  In  addition,  CAHPS 
surveys  include  a  free,  easy-to-use  CAHPS  Survey  and  Reporting  Kit.  The  kit  contains  a  set  of 
mail  and  telephone  survey  questionnaires  including  Spanish  language  versions,  sample  reporting 
formats,  and  a  handbook  with  step-by-step  instructions  for  sampling,  administration,  analysis, 
and  reporting.  The  handbook's  instructions  are  easy  to  understand  and  allow  flexibility  within 
the  sample  and  analysis  design.  The  handbook  also  includes  a  comprehensive,  statistical 
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software  package  as  well  as  the  telephone  number  and  e-mail  address  of  the  technical  assistance 
hotline. 

There  are  a  wide  variety  of  other  existing  questionnaires  that  can  be  selected  as  the  survey 
instrument.  While  some  have  undergone  extensive  reliability  and  validity  testing,  others  have 
beefi  developed  with  little  or  no  testing  for  reliability  and  validity.  Unless  there  is  evidence  of  an 
instrument's  reliability  and  validity  testing  and  results,  an  instrument's  reliability  and  validity 
should  always  be  suspect.  Even  pre-existing,  validated  survey  instruments  may  not  have  been 
validated  in  a  Medicaid  population.  Instruments  not  validated  in  a  Medicaid  population  may  not 
yield  valid  or  reliable  results  for  this  population. 

Another  advantage  of  existing  instruments  is  that  use  of  the  same  questionnaire,  methods,  and 
analysis  for  surveys  of  MCOs/PHPs,  population  groups,  and  States,  can  permit  a  particular 
survey's  findings  to  be  compared  against  those  of  other  studies.  For  example,  standardization  of 
the  survey  instruments,  methodology,  and  report  formats  in  CAHPS  provides  reliable  and  valid 
measures  that  can  be  compared  across  MCOs/PHPs,  across  MCOs/PHPs  and  fee-for-service 
(FFS),  as  well  as  across  a  wide  range  of  respondents  and  States. 

When  an  existing  survey  instrument  was  used,  the  EQRO  should  document  the  extent  of 
reliability  and  validity  testing  of  the  survey  instrument. 

New  survey  instruments.  In  some  cases,  a  survey  sponsor  may  have  chosen  to  create  a  new 
survey  that  was  developed  specifically  for  a  particular  survey.  Usually,  new  survey  instruments 
are  developed  and  used  when  the  purposes  and  objectives  of  the  study  require  answers  to 
questions  that  are  not  addressed  by  existing  instruments.  Such  survey  instruments  can  capture 
information  that  is  of  great  interest  and  relevance  to  the  questions  under  study.  However, 
assuring  the  reliability  and  validity  of  new  surveys  is  costly  and  takes  time.  Without  such 
reliability  and  validity  testing,  surveys  can  have  serious  methodological  shortcomings  and  be 
vulnerable  to  criticism.  A  1997  report  by  the  Department  of  Health  and  Human  Services  (DHHS) 
Inspector  General,  "Medicaid  Managed  Care:  The  Use  of  Surveys  as  a  Beneficiary  Protection 
Tool"  found  that  some  State-specific  surveys  were  designed  to  meet  too  many  objectives  and 
provided  little  useful  data.- When  external  quality  review  is  to  incorporate  the  findings  of  a  new 
survey  instrument  and  methodology,  the  EQRO  should  document  the  extent  of  reliability  and 
validity  testing  of  the  survey  instrument  and  methodology.  The  results  of  the  reliability  and 
validity  testing  are  an  important  consideration  in  determining  the  extent  to  which  the  EQRO 
incorporates  the  survey  findings  into  its  analysis  and  evaluation  of  information  on  the  access, 
quality  and  timeliness  of  health  care  services  furnished  to  Medicaid  beneficiaries  enrolled  in 
MCOs/PHPs. 

Adaptations  of  existing  surveys.  Survey  sponsors  also  may  have  decided  to  adapt  an  existing 
questionnaire  by  adding  or  deleting  items,  modifying  questions,  or  using  only  certain  groups  of 
questions  to  respond  to  a  specific  issue  under  study.  This  approach  provides  flexibility  to  add  or 
change  the  data  to  be  collected,  while  providing  many  of  the  advantages  of  using  a  pre-existing 
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questionnaire.  However,  this  approach  also  can  raise  questions  about  the  validity  and  reliability 
of  the  new  or  modified  questions,  as  well  as  the  survey  overall.  Validated  instruments  are  tested 
"as  a  whole,"  and  any  modifications  can  change  the  focus  and  purpose  of  the  tool,  and  thereby 
the  validity  and  reliability  of  the  tool  as  a  whole  and/or  its  individual  questions.  Adapting  an 
existing  survey  is  easier  if  CAHPS  is  used,  as  the  CAHPS  survey  instruments  have  been 
specifically  designed  with  opportunities  to  customize  the  questionnaire  by  adding  questions  from 
a  list  of  supplemental  optional  items,  as  well  as  State-designed  questions.  Most  pre-existing 
questionnaires  are  not  designed  to  accommodate  such  modifications  and  require  additional 
testing.  Modifications  and  additional  testing  are  time  intensive  and  will  likely  result  in  increased 
costs. 

The  EQRO  should  review  the  evidence  about  the  reliability  and  validity  testing  of  all  survey 
instruments,  regardless  of  whether  they  are  existing,  new,  or  adapted  instruments.  Each  survey's 
reliability  and  validity  testing  should  be  an  important  element  in  determining  to  what  extent  the 
EQRO  relies  upon  the  survey  findings  to  inform  its  analysis  and  evaluation  of  access,  quality  and 
timeliness  of  health  care. 


ACTIVITY  4:        Review  the  sampling  plan. 

When  a  survey  sponsor  decides  to  collect  information  directly  from  beneficiaries,  providers,  or 
other  unit  of  analysis,  it  must  decide  how  may  "units  "  will  need  to  be  surveyed.  A  survey 
sponsor  could  decide  to  take  a  "census; "  i.e.,  ask  every  individual  entity  which  the  survey  is 
designed  to  represent,  to  answer  the  survey  questions.  For  example,  a  survey  sponsor  could 
decide  to  survey  all  Medicaid  beneficiaries  or  all  Medicaid  participating  providers.  However, 
because  the  cost  of  contacting  all  these  entities  is  expensive,  surveys  typically  are  administered  to 
only  a  portion  (a  "sample")  of  the  target  population.  Whenever  a  sample,  as  opposed  to  a  census, 
is  used,  it  is  very  important  to  select  the  sample  members  in  a  way  that  represents  as  closely  as 
possible  the  population  about  whom  information  is  desired  -  both  those  included  in  the  sample 
and  those  not  selected  to  respond  to  the  survey.  The  EQRO  should  review  documentation  related 
to  the  sampling  plan,  particularly  (1)  the  definition  of  the  study  population;  (2)  specifications  for 
the  sample  frame;  (3)  the  type  of  sampling  used;  (4)  the  adequacy  of  the  sample  size;  and  (5) 
sample  selection  procedures.  The  level  of  detail  involved  in  this  review  will  require  that  the 
EQRO  use  professional  statisticians.  The  EQRO  will  need  to  make  a  judgement  about  whether 
the  sample  selected  was  sufficiently  representative  of  the  study  population  for  the  EQRO  to  have 
confidence  in  the  survey  findings. 

Step  1:  Identify  the  study  population. 

The  EQRO  should  first  document  what  population  the  survey  was  designed  to  study;  e.g.,  all 
Medicaid  beneficiaries  enrolled  in  managed  care,  or  all  children  with  special  health  care  needs. 
The  "population  "  represents  all  units  of  analysis  (sometimes  referred  to  as  "the  universe")  about 
which  the  survey  questions  are  raised.  A  good  sample  is  a  small  version  of  the  population  to  be 
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studied  and  is  representative  of  the  population  in  terms  of  certain  pre-identified  characteristics 
(e.g.,  race,  age,  MCO/PHP  affiliation.) 


Step  2:  Review  the  sample  frame. 

Once  the  target  population  for  the  survey  is  identified,  the  EQRO  must  also  identify  and 
understand  the  construction  of  the  survey's  "sample  frame.  "  The  "sample  frame"  is  a  listing  of 
all  members  of  the  study  population  who  are  eligible  for  the  study.  From  this  listing,  the  sample 
is  selected.  Questions  used  to  define  individuals  eligible  for  the  survey  (the  sample  frame)  might 
include,  for  example: 

Was  a  minimum  period  of  enrollment  in  a  MCO/PHP  necessary  for  individuals  who  were 
surveyed?  Did  beneficiaries  have  to  be  continuously  enrolled  in  a  specific  MCO/PHP  for 
a  specified  period  of  time  to  be  included  in  the  survey?  Frequently,  a  minimum  period  of 
continuous  enrollment,  such  as  six  months,  is  required  to  increase  the  likelihood  that 
enrollees  have  had  contact  with  their  specific  MCO/PHP. 

Were  any  MCO  enrollees  excluded?  Why?  For  example,  if  a  study  aimed  to  address 
children  with  special  health  care  needs,  was  it  limited  only  to  children  receiving  SSI  or 
Title  V,  or  did  it  address  all  children  that  a  MCO/PHP  might  identify  as  having  special 
health  care  needs,  or  all  children  whose  family  might  identify  them  as  having  a  special 
health  care  needs? 

Was  only  one  member  per  household  allowed  to  be  surveyed? 
Were  children  allowed  to  be  surveyed? 

When  was  open  enrollment  and  how  did  it  impact  survey  administration? 
Step  3:  Review  the  type  of  sampling  used. 

There  are  two  basic  types  of  statistical  sampling:  probability  sampling  and  nonprobability 
sampling.  Probability  sampling  leaves  selection  of  the  units  to  be  included  in  the  survey  up  to 
chance.  It  is  designed  to  prevent  any  biases  in  the  sample  selected  by  ensuring  that  each  segment 
of  the  study  population  is  represented  in  the  sample  to  the  extent  it  composes  part  of  the 
population  under  study.  Simple  random  sampling  is  the  most  common  form  of  probability 
sampling  used  in  survey  research.  Population  members  are  generally  assigned  a  number,  and 
random  numbers  generated  by  a  computer  or  table  of  random  numbers  are  used  to  select 
members  from  the  population.  This  sampling  approach  ensures  that  all  members  of  the  target 
population  have  an  equal  chance  to  be  selected.  The  sample  is  thus  assumed  to  be  fully 
representative  of  the  population. 
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Stratified  random  sampling  also  may  be  used.  This  technique  calls  for  dividing  the  population 
(on  the  basis  of  prior  knowledge  about  the  population)  into  specific,  pre-identified  strata  or 
subgroups  that  are  homogeneous  within  themselves  with  respect  to  certain  characteristics  (e.g., 
ethnicity,  age,  diagnoses).  A  random  sample  is  then  taken  from  each  stratum  or  subgroup. 
Stratification  is  done  both  to  improve  the  accuracy  of  estimating  the  total  population's 
characteristics  and  to  provide  information  about  the  characteristics  of  interest  within  subgroups. 
However,  stratified  random  sampling  requires  more  information  about  the  population,  and  can 
also  require  a  larger  total  sample  size.  As  a  result,  it  is  typically  more  expensive  than  simple 
random  sampling.  Stratified  sampling  may  also  involve  "weighting"  the  sample.  In  this  process,  a 
survey  would  select  a  disproportionately  larger  number  of  units  of  analysis  from  one  or  more  of 
the  strata  to  allow  the  survey  to  produce  information  on  that  particular  strata;  e.g.,  individuals 
dually  receiving  both  Medicare  and  Medicaid. 

Non-probability  sampling  methods  are  based  on  the  decisions  of  those  administering  the  survey, 
and  not  on  random  chance.  For  example,  a  State  survey  sponsor  might  wish  to  obtain  information 
about  new  Medicaid  beneficiaries  by  surveying  all  new  beneficiaries  eligible  in  a  particular 
month.  It  might  wish  to  obtain  information  on  average  stage  of  pregnancy  at  time  of  the 
establishment  of  Medicaid  eligibility  by  examining  the  records  of  women  giving  birth  at  a 
particular  hospital.  In  such  cases,  the  information  obtained  by  the  State  will  have  been  biased. 
Non-random  sampling  methods  such  as  these  also  do  not  lend  themselves  to  statistical  analysis. 
Because  of  the  risk  of  biased  results  and  the  obstacles  to  statistical  analysis,  nonprobability 
sampling  is  discouraged.  However,  at  times  it  can  be  an  appropriate  and  efficient  way  of 
collecting  needed  information. 

Step  4:  Review  adequacy  of  sample  size. 

A  sample  only  collects  information  from  a  fraction  of  the  population.  Based  on  data  obtained 
from  the  sample,  estimates  are  produced  about  the  utilization  of  health  care  services,  satisfaction 
with  care,  quality  of  care  or  other  issues.  However,  these  estimates  are  at  risk  of  being 
"inaccurate  "  to  the  extent  that:  (1)  they  are  different  from  the  population's  true  values  because 
only  a  portion  of  the  entire  population  was  involved  in  producing  the  estimates,  and  (2)  a 
different  sample  will  give  different  estimates.  Fortunately,  the  risk  of  inaccuracy  can  be  reduced 
by  carefully  selecting  the  sample  size. 

Two  factors  influence  the  determination  of  the  appropriate  sample  size  for  a  survey3:  (1)  the  level 
of  certainty  desired  for  the  estimate  to  be  produced  and  2)  the  margin  of  error  that  is  defined  as 
acceptable.  Combining  these  two  factors  establishes  the  parameters  for  the  estimates  to  be 
produced  by  the  survey  (e.g.,  a  survey  would  target  the  answers  to  its  questions  to  be  95% 


A  third  factor  (the  true  frequency  of  the  occurrence  of  a  characteristic  within  a  population)  is  also  often 
used  to  calculate  the  sample  size  for  some  studies.  However,  since  a  survey  studies  multiple  questions,  and  the  true 
prevalence  of  responses  to  any  of  these  questions  may  be  variable  across  questions,  and  change  over  time,  the  most 
prudent  course  of  action  is  to  assume  the  need  for  the  maximum  sample  size  by  assuming  that  the  presence  or 
absence  of  the  characteristic  in  question  is  evenly  distributed  across  the  population. 
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accurate  with  a  5%  margin  of  error).  The  greater  the  certainty  and  the  smaller  the  margin  of  error 
desired,  the  larger  the  sample  will  need  to  be  (and  the  more  expensive  the  survey  will  be).  In 
addition,  oversampling  (sampling  for  a  greater  number  of  entities  than  are  actually  needed  to 
produce  valid  estimates)  is  sometimes  undertaken,  to  compensate  for  an  anticipated  percent  of 
non-respondents  to  the  survey.  Because  of  these  and  other  factors  (e.g.,  whether  the  State 
intended  the  survey  to  provide  estimates  of  individual  MCO/PHP  performance  or  whether  the 
survey  findings  were  intended  to  compare  MCP/PHP  performance),  in  order  to  review  the 
adequacy  of  the  sample  size,  the  EQRO  needs  to  evaluate:  ( 1 )  the  level  of  certainty  desired  about 
the  estimate  to  be  produced  and  2)  the  margin  of  error  that  was  defined  as  acceptable.  The  EQRO 
will  need  a  qualified  statistician  to  review  the  adequacy  of  the  sample  size  that  was  used  for  the 
survey. 

If  the  CAHPS  survey  is  used,  the  EQRO  can  take  advantage  of  the  statistical  analysis  already 
performed  by  the  CAHPS'  developers.  However,  these  will  differ  from  the  specifications 
provided  by  the  National  Committee  for  Quality  Assurance  (NCQA)  in  the  CAHPS  2. OH 
manual.  CAHPS  2. OH  is  an  adaptation  of  CAHPS  developed  by  the  NCQA,  in  conjunction  with 
AHCPR,  for  use  with  NCQA's  Healthplan  Employer  Data  and  Information  Set  -  HEDIS.  The 
CAHPS  2. OH  questionnaire  is  identical  to  the  CAHPS  questionnaire  except  for  the  inclusion  of 
the  HEDIS  smoking  cessation  counseling  questions.  This  difference  produces  different  sampling 
specifications  for  CAHPS  2.0  and  for  NCQA's  CAHPS  2.0H  version.  NCQA's  CAHPS  2.0H 
requires  a  larger  sample  size  in  order  to  obtain  adequate  numbers  of  respondents  who  can  answer 
the  smoking  questions. 

Step  5:  Review  the  sample  selection  procedures. 

The  EQRO  will  review  the  sample  selection  procedures.  This  includes  reviewing  the  statistical 
program  or  other  process  used  to  generate  the  sample.  The  EQRO  will  need  to  determine  the 
extent  to  which  the  selection  of  sample  members  from  the  sample  frame  was  conducted  in  a 
manner  that  protects  against  bias. 

ACTIVITY  5:  Review  the  adequacy  of  the  response  rate. 

The  EQRO  must  assess  the  adequacy  of  the  response  rate.  This  includes  a  review  of :  1 )  how 
respondents  in  the  sample  were  contacted;  and,  2)  what  the  response  rate  was  and  how  it  was 
calculated. 

Step  1:  Review  the  strategy  for  contacting  individuals  participating  in  the  survey. 

Surveys  can  be  administered  in  a  face-to-face  interview,  by  telephone  interview,  by  a  mailed 
questionnaire  to  be  completed  by  the  respondent  and  mailed  back,  or  by  a  combination  of  these 
approaches.  Because  of  the  importance  of  achieving  a  high  response  rate,  considerable  research 
has  been  conducted  on  these  approaches  to  survey  administration.  Recent  studies  indicate  that  the 
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percentage  of  nonrespondents  is  increasing,  making  survey  research  more  difficult.  Although  in 
the  past  survey  researchers  believed  that  face-to-face  interviewing  produced  higher  response 
rates,  recent  research  challenges  this  view.  No  single  method  of  survey  administration  is 
believed  to  be  superior  in  all  circumstances. 

Step  2:  Review  the  response  rate. 

The  response  rate  generally  is  the  number  of  entities  that  respond  to  the  survey  divided  by  the 
number  of  entities  selected  to  respond  to  the  survey.  However,  the  exact  definition  is  generally 
more  detailed  and  complex  that  this;  there  are  a  number  of  different  ways  to  calculate  response 
rates,  as  well  as  different  strategies  for  reporting  these  rates.  In  addition,  the  definition  of 
response  rate  may  differ  depending  on  mode  of  administration.  For  example,  mail  surveys 
typically  may  exclude  non-deliverable  addresses  from  the  denominator,  while  telephone 
interviews  may  exclude  from  the  denominator  people  with  non-working  telephone  numbers  and 
those  who  no  longer  reside  in  the  household.  The  response  rate  is  important  because  a  low  rate 
reduces  the  ability  to  make  accurate  conclusions  (statistical  inferences)  about  the  population 
under  study.  Those  individuals  who  do  not  respond  to  the  survey  may  differ  systematically  from 
individuals  who  respond,  thereby  biasing  the  results. 

Medicaid  beneficiaries  typically  have  a  low  survey  response  rate.  Medicaid  populations  tend  to 
have  lower  literacy  levels  than  average,  are  more  likely  to  speak  primary  languages  other  than 
English,  are  highly  mobile,  and  have  high  rates  of  inaccurate  or  unavailable  telephone  numbers 
and  mailing  addresses.  As  a  result,  surveys  of  Medicaid  beneficiaries  tend  to  require  extra  effort 
to  maximize  response  rates.  These  efforts  can  include  using  an  expansive  contact  strategy  that 
emphasizes  tracking  methods,  and  a  mixed-mode  approach  to  survey  administration  that  typically 
combines  mail  and  telephone  survey  procedures.  However,  even  with  these  additional  measures, 
response  rates  for  Medicaid  beneficiaries  are  likely  to  be  lower  than  for  many  other  populations. 
The  CAHPS  developers  suggest  that  the  target  response  rate  for  administering  CAHPS  to 
Medicaid  beneficiaries  should  range  from  40%  to  60%.  They  also  recommend  that  survey 
vendors  focus  on  strategies  that  promote  high  response  rates  and  develop  a  plan  of  correction 
actions  to  take  if  the  response  rate  falls  short  of  the  goal. 

The  EQRO  should  evaluate  the  response  rate  and  discuss  with  survey  sponsors  the  potential 
reasons  for  non-response,  and  the  extent  to  which  nonresponse  may  have  introduced  bias  in  the 
survey  findings.  The  EQRO  also  should  examine  the  specific  methods  for  calculating  both  raw 
and  adjusted  response  rates.  For  example: 

Raw  response  rates  can  be  calculated  by  : 

 Number  of  completed  surveys   

Total  number  of  targeted  survey  respondents 
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Adjusted  response  rates  can  be  calculated  by: 

 Number  of  completed  surveys  

Total  number  of  targeted  survey  respondents  minus  deceased  and  ineligible  due  to 
hospitalization,  cognitive  impairment,  or  other  reasons  specified  in  the  methodology) 

The  EQRO  will  need  to  assess  the  response  rate,  whether  the  survey  employed  a  reasonable 
method  for  calculating  the  response  rate,  potential  sources  of  non-response  and  bias,  and  the 
extent  to  which  the  response  rate  weakens  or  strengthens  the  generalizeability  of  the  survey 
findings. 

ACTIVITY  6:  Review  survey  data  analysis  and  findings/conclusions. 

The  EQRO  should  review  the  approach  to  analyzing  the  survey  data  and  the  findings/conclusions 
of  the  survey. 

Data  quality  control.  The  EQRO  should  review  how  the  survey  sponsor  handled  responses  that 
failed  edit  checks,  the  treatment  of  missing  data,  and  procedures  for  determination  of 
usable/complete  surveys. 

Data  analysis.  The  EQRO  should  review  how  the  survey  sponsor  analyzed  the  survey  data.  Each 
MCO/PHP  should  have  been  treated  as  the  basic  unit  of  analysis  and  reporting.  The  analysis 
should  include  simple  statistical  procedures  such  as  measures  of  central  tendency  and  frequency 
distributions  for  specific  survey  questions.  In  addition,  the  findings  may  include  a  statistical 
analysis  of  the  extent  to  which  there  are  differences  in  responses  to  questions  according  to  survey 
subgroups  (e.g.,  MCOs/PHPs).  Any  differences  should  be  examined  using  standard  statistical 
tests  such  as  the  F-test  and  the  T-test.  The  F-test  may  be  used  to  determine  if  the  average  scores 
of  any  entity  (e.g.,  MCO  or  PHP)  differ  significantly  from  that  of  other  MCOs/PHPs.  The  F-test 
tests  the  hypothesis  that  all  MCO/PHP  means  (averages)  are  equal  by  comparing  the  relative  size 
of  two  variances.  The  F-test  will  be  significant  if  at  least  one  MCO/PHP  mean  is  statistically 
different  from  any  other  MCO/PHP  mean.  If  the  F-test  indicates  that  there  are  significant 
differences,  a  series  of  t-tests  may  be  conducted  to  identify  which  MCO/PHP  is  different  from 
other  MCO/PHPs.  The  t-test  tests  the  hypothesis  that  two  specific  MCO/PHP  means  are 
significantly  different  from  each  other.  A  significant  t-test  indicates  that  the  two  MCOs/PHPs 
represent  two  populations  with  different  means. 

In  addition,  multivariate  regression  and/or  analysis  of  co variance  might  have  been  used  to  further 
identify  meaningful  differences  between  MCOs/PHPs  while  controlling  for  respondent 
characteristics.  For  example,  a  driver  analysis  and  a  quadrant  analysis  may  have  been  used  to 
gain  more  insight  into  the  survey  measures  and  possible  approaches  to  future  improvement. 
Driver  analysis  refers  to  statistical  techniques  that  may  be  used  to  identify  the  important  factors 
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or  "drivers"  that  significantly  affect  some  outcome  measure.  In  this  context,  the  driver  analysis 
will  identify  the  most  important  factors  (drivers)  that  significantly  affect  overall  scores. 
Individual  questions  can  be  analyzed  to  determine  their  effect  on  overall  scores.  These  identified 
drivers,  along  with  their  current  measures  will  then  be  subjected  to  a  quadrant  analysis  which 
compares  the  importance  of  the  drivers  against  the  current  survey  measures  on  these  drivers.  For 
example,  if  access  to  specialists  has  been  identified  as  a  driver  of  patient  satisfaction,  then 
depending  on  the  importance  of  access  as  a  driver  and  the  current  performance  on  access,  there 
could  be  four  different  combinations  or  quadrants:  (1)  high  importance  and  high  performance,  (2) 
high  importance  and  low  performance,  (3)  low  importance  and  low  performance,  and  (4)  low 
importance  and  high  performance.  High  importance  means  that  access  to  specialists  has  an 
important  effect  on  satisfaction,  and  high  performance  means  that  access  has  been  appropriately 
provided.  Therefore,  quadrant  (1)  identifies  strength,  i.e.  satisfactory  performance  on  a  crucial 
driver.  Similarly,  quadrant  (2)  identifies  areas  needing  improvement;  quadrant  (3)  and  (4)  are 
areas  of  low  leverage  and  insignificance.  The  most  important  insight  that  can  be  gained  through 
quadrant  analysis  is  the  identification  of  improvement  opportunities.  MCO/PHPs  can  then  make 
strategic  decisions  about  where  to  devote  their  efforts  and  resources  in  order  to  improve 
MCO/PHP  scores. 

Consistent  with  the  purposes  and  objectives  of  the  survey,  populations  within  each  MCO/PHP 
could  also  be  analyzed  and  reported  separately.  For  instance,  the  State  may  be  interested  in 
whether  any  particular  finding  differs  significantly  across  geographic  locations,  demographic 
groups,  socio-economic  groups,  or  other  identifiable  subgroups  (e.g.,  frequent  and  infrequent 
users  of  health  care  services).  Survey  results  also  may  have  been  adjusted  for  external  factors  that 
influence  results;  e.g.,  studies  suggest  that  age,  education,  and  self-reported  health  status,  among 
others,  have  significant  effect  on  patient  satisfaction. 

Regardless  of  the  statistical  analyses  performed,  the  data  analysis  should  yield  clear  information 
about  the  performance  of  each  MCO/PHP  and,  as  appropriate,  differences  among  MCOs/PHPs. 

ACTIVITY  7:  Document  evaluation  of  survey 

Using  the  information  obtained  from  activities  1  -  6,  including  the  review  of: 

survey  purpose,  objectives  and  intended  audience(s); 

•  reliability  and  validity  of  the  survey  instrument; 

•  adequacy  of  the  sampling  strategy  and  response  rate;  and 
data  obtained  and  its  analysis, 

the  EQRO  should  make  a  determination  of  the  generalizeability  of  the  survey  findings,  and  the 
extent  to  which  the  survey  findings  can  be  relied  upon  to  make  inferences  about  the  access, 
quality,  and  timeliness  of  health  care  delivered  to  Medicaid  beneficiaries  enrolled  in 
MCOs/PHPs.  The  EQRO  should  document  these  conclusions  and  provide  written  findings  on: 
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1)  the  survey's  technical  strengths  and  weaknesses;  2)  the  limitations  /generalizeability  of  survey 
findings;  3)  conclusions  drawn  from  the  survey  data;  4)  a  detailed  assessment  of  each 
MCO/PHP's  strengths  and  weaknesses  with  respect  to  (as  appropriate)  access,  quality,  and/or 
timeliness  of  health  care  furnished  to  Medicaid  enrollees;  and  5)  as  the  State  determines 
methodologically  appropriate,  comparative  information  about  all  MCOs/PHPs. 
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Vasudha  Narayanan  is  a  study  Director  with  a  diverse  background  in  market  analysis, 
consumer  interface,  and  project  management.  At  WEST  AT,  she  has  been  responsible  for 
questionnaire  design,  data  collection  and  the  design  and  management  of  every  aspect  of 
survey  operations,  with  particular  emphasis  on  studies  of  health  insurance.  Ms. 
Narayanan's  relevant  project  experience  with  WESTAT  includes  participation  with:  a 
survey  of  experiences  with  Medicare  HMO's  conducted  for  the  Kaiser  Family 
Foundation;  a  health  communication  market  research  for  providers  task  order  conducted 
to  help  the  Health  Care  Financing  Administration  understand  the  information  needs  of 
providers  and  other  partners;  and  a  beneficiary  market  research  task  order  conducted  for 
the  Health  Care  Financing  Administration  which  included  an  inventory  of  existing 
information  on  the  information  needs  of,  and  how  to  communicate  with,  Medicare 
beneficiaries  on  the  Medicare  program,  supplemental  health  insurance  and  managed  care, 
and  general  health  topics. 
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John  Neff,  Professor  of  Pediatrics  at  the  University  of  Washington  School  of  Medicine 
and  Director  of  the  Center  for  Childhood  Chronic  Conditions  at  Children's  Hospital  and 
Regional  Medical  Center  in  Seattle,  Washington,  is  a  graduate  of  Pomona  College  and 
Harvard  Medical  School.  He  was  on  the  faculty  at  Johns  Hopkins  Medical  School  from 
1968  to  1981,  and  served  in  the  Dean's  Office  of  the  Medical  School  from  1968-1975. 
From  1975  to  1981  he  directed  the  pediatric  program  at  The  Baltimore  City  Hospitals, 
where  he  developed  a  pre-paid  practice  plan  for  the  city's  foster  children. 

From  1981  to  1988,  he  was  Medical  Director  and  Vice  President  of  Children's  Hospital 
and  Regional  Medical  Center  in  Seattle  and  Associate  Dean  at  the  University  of 
Washington  School  of  Medicine.  In  1998  he  became  Director  of  the  Center  for  Children 
with  Special  Needs  and  Chronic  Health  Condition. 

He  has  served  on  the  State  Perinatal  Advisory  Committee,  the  State  Parent/Child  Health 
Advisory  Council,  and  The  Family  and  Children's  Ombudsman  Committee.  Since  1933 
he  has  chaired  the  Washington  State  Issue  Investigative  Group  on  Children  with  Special 
Health  Care  Needs  and  he  has  consulted  with  the  National  Association  of  Children's 
Hospitals  and  Related  Institutions  (NCDHRI)  on  defining  children  with  special  needs  and 
their  capitation  needs  in  managed  care. 

He  has  broad  interests  in  preventive  medicine,  child  health,  and  children  with  special  * 
health  care  needs. 
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Dr.  Bethell,  is  the  Senior  Vice  President,  Research  and  Policy  at  FACCT—  The  Foundation 
for  Accountability.  FACCT  is  a  not-for-profit  organization  dedicated  to  helping  consumers 
make  better  health  care  decisions.  Dr.  Bethell  has  more  than  15  years  of  experience  in  both 
health  care  policy  and  administration  and  public  health  at  the  state,  local  and  federal  levels. 
She  is  a  former  senior  policy  analyst  with  VHA  Inc.  in  Washington,  DC,  senior  research 
associate  with  Chicago's  Rush  Primary  Care  Institute  and  health  policy  analyst  with  both  the 
American  Association  of  Retired  Persons  and  California's  Health  Access  Foundation.  In 
each  of  these  roles,  her  primary  focus  has  been  on  shaping  a  health  care  system  that  is 
organized,  financed  and  evaluated  in  ways  that  meet  the  health  needs  of  the  public.  Dr. 
Bethell  also  has  extensive  experience  working  with  communities  to  assess  health  needs  and 
develop  partnerships  to  improve  health.  She  holds  a  Ph.D.  in  health  services  and  policy 
research  from  the  University  of  California  at  Berkeley.  Her  research  has  focused  on  both 
the  role  of  the  health  care  system  in  improving  community  health  and  the  development  and 
testing  of  indicators  of  performance  for  the  health  care  system  in  the  area  of  prevention  and 
primary' care. 

Dr.  Bethell,  is  the  Senior  Vice  President,  Research  and  Policy  at  FACCT—  The  Foundation 
for  Accountability.  Since  FACCT's  beginning,  she  has  been  responsible  for  the  development 
of  quality  performance  measures  and  consumer  research  and  is  director  of  the  Children  and 
Adolescent  Health  Measurement  Initiative  -  a  national  collaborative  between  FACCT  and 
the  National  Committee  for  Quality  Assurance. 


VALIDATING  PERFORMANCE  MEASURES 


I.        PURPOSE  OF  THE  PROTOCOL 

This  protocol  specifies  activities  to  be  undertaken  by  an  EQRO1  in  order  to  validly: 

1 .  Evaluate  the  accuracy  of  Medicaid  performance  measures  reported  by,  or  on  behalf  of,  a 
Managed  Care  Organization  (MCO)  or  a  Prepaid  Health  Plan  (PHP);  and 

2.  Determine  the  extent  to  which  Medicaid-specific  performance  measures  calculated  by  an 
MCO/PHP  (or  by  an  entity  acting  on  behalf  of  an  MCO  or  PHP)  followed  specifications 
established  by  the  State  Medicaid  agency  (the  State)  for  the  calculation  of  the 
performance  measure(s). 

II.       ORIGIN  OF  THE  PROTOCOL 

This  protocol  was  derived  from  protocols  and  tools  commonly  used  in  the  public  and  private- 
sectors  for  auditing  performance  measures.  These  include: 

the  National  Committee  for  Quality  Assurance's  (NCQA)  1 999  HEDIS® 
publication:  Volume  5,  HEDIS  Compliance  Audit™  Standards  and  Guidelines; 

•  tools  used  by  the  Island  Peer  Review  Organization  (IPRO)  in  their  audits  of 
HEDIS  measures  for  Medicare;  and 

•  documents  from  the  MEDSTAT  Group,  Inc.,  published  in  conjunction  with  work 
performed  for  HCFA  in  1997  and  1998. 

A  review  of  these  tools  found  that,  while  there  were  differences,  these  documents  had  much  in 
common. 

Both  NCQA's  and  IPRO's  documents  address  the  validation  of  HEDlS  measures  only.  They 
assess: 

•  the  structure  and  integrity  of  the  MCO's/PHP's  underlying  information  system 
(IS); 

•  MCO/PHP  ability  to  collect  valid  data  from  various  internal  and  external  sources; 

•  vendor  (or  subcontractor)  data  and  processes,  and  the  relationship  of  these  data 


It  is  recognized  that  a  State  Medicaid  agency  may  choose  an  organization  other  than  an  EQRO  as  defined 
in  Federal  regulation  to  validate  Medicaid  performance  measures  submitted  by  or  on  behalf  of  an  MCO/PHP. 
However,  for  convenience,  in  this  protocol  we  use  the  term,  "external  quality  review  organization"  (EQRO)  to  refer 
to  any  organization  that  validates  performance  measures. 
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sources  to  those  of  the  MCO/PHP; 

•  MCO/PHP  ability  to  integrate  different  types  of  information  from  disparate  data 
sources  into  a  data  repository  or  set  of  consolidated  files  for  use  in  constructing 
plan-level  performance  measures;  and 

•  documentation  of  the  MCO's/PHP's  processes  to:  collect  appropriate  and  accurate 
data,  manipulate  those  data  through  programmed  computer  queries,  internally 
validate  the  results  of  the  operations  performed  on  the  data  sets,  follow  specified 
procedures  for  calculating  the  specified  performance  measures,  and  report  the 
measures  appropriately. 

The  MEDSTAT  publications  focus  primarily  on  validation  of  encounter-level  data,  and  the  use 
of  those  data  in  Medicaid  MCO  performance  measures,  regardless  of  whether  the  performance 
measures  are  based  on  the  NCQA  Medicaid  HEDIS  measures  or  have  been  developed  by  other 
groups  or  organizations.  However,  the  MEDSTAT  publications  do  not  provide  detailed 
instructions  or  guidelines  that  an  EQRO  might  use  to  validate  the  plan-level  performance 
measures  once  the  encounter  data  are  validated. 

The  protocol  presented  here  is  consistent  with  the  approaches  used  in  the  IPRO  and  NCQA 
documents,  but  is  designed  with  a  MEDSTAT-like  approach  in  that  it  describes  how  to  validate 
all  performance  measures  -  HEDIS  measures  as  well  as  non-HEDIS  measures.  It  varies  from  the 
IPRO  and  NCQA  protocols  in  that  certain  components  of  performance  measure  validation  may 
be  performed  as  a  part  of  this  protocol  or  accomplished  through  some  other  mechanism(s)  used 
by  the  State.  For  example,  as  part  of  this  protocol,  an  assessment  of  the  MCO's/PHP's  IS  is 
required.  This  IS  assessment  may  be  conducted  as  a  part  of  this  protocol  by  the  EQRO  validating 
the  performance  measures,  or  the  EQRO  may  review  an  assessment  of  the  MCO's  /PHP's  IS 
conducted  by  another  party. 


HI.      OVERVIEW  OF  THE  PROTOCOL 

The  protocol  assumes  that  the  State  has  specified: 

the  performance  measures  to  be  calculated  by  MCOs/PHPs; 

•  the  specifications  to  be  followed  in  calculating  these  measures;  and 

•  the  manner  and  mechanisms  for  reporting  these  measures  to  the  State. 

Protocol  activities  address: 

1 .  Review  of  the  data  management  processes  of  the  MCO/PHP; 

2.  Evaluation  of  algorithmic  compliance  (the  translation  of  captured  data  into  actual 
statistics)  with  specifications  defined  by  the  State;  and 

3.  Verification  of  either  the  entire  set  or  a  sample  of  the  State-specified  performance 
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measures  to  confirm  that  the  reported  results  are  based  on  accurate  source  information. 

The  protocol  consists  of  three  phases  of  activities:  Pre-Onsite,  Onsite  and  Post-Onsite  activities. 
For  each  of  these  phases,  the  protocol  specifies:  1)  outcomes  or  objectives;  2)  activities  to  be 
performed;  3)  methods  of  evaluation;  and  4)  tools  and  worksheets. 

Pre-Onsite  activities  involve: 

1 .  Communicating  with  the  State  to  ensure  that  the  EQRO  understands: 

•  the  measures  to  be  validated  (i.e.,  the  entire  set  versus  a  subset  of  those  calculated 
by  the  MCO/PHP);  and 

•  the  methodology(ies)  the  State  requires  the  MCO/PHP  to  follow  when  calculating 
and  reporting  the  performance  measures;  and 

2.  Either  conducting  an  assessment,  or  reviewing  the  results  of  a  prior  assessment,  of  the 
MCO's/PHP's  underlying  IS. 

Onsite  activities  focus  on:  1)  following  up  on  IS  findings  identified  in  the  Pre-Onsite  activities  as 
being  potentially  problematic  or  in  need  of  further  review  or  clarification;  and  2)  validating  the 
production  and  reporting  of  performance  measures  through  observation  of  documentation  or 
procedures.  These  activities  include: 

1 .  Reviewing  and  assessing  the  procedures  the  MCO/PHP  has  in  place  for  collecting  and 
integrating  medical,  financial,  member  and  provider  information,  covering  both  clinical 
and  service-related  data,  from  internal  and  external  sources; 

2.  Evaluating  processes  used  by  the  MCO/PHP  to  produce  performance  measures;  e.g., 
sampling,  calculating  denominators  and  numerators;  and 

3.  Evaluating  the  MCO's/PHP's  processes  for  reporting  required  performance  measures  to 
the  State. 

To  accomplish  these  activities,  the  EQRO  reviews  policy  and  procedure  manuals  and  documents, 
observes  required  activities,  and  conducts  interviews  with  key  MCO/PHP  staff  such  as  the 
Director  of  Health/Medical  Information  Systems,  IS  programmers  or  operators,  Director  of 
Member/Patient  Services,  Director  of  Utilization  Management,  and  the  Director  of  Quality 
Improvement. 

Post-Onsite  activities  focus  on  the  analysis  of  the  data  and  information  obtained  through  Pre- 
Onsite  and  Onsite  activities,  and  submission  of  the  validation  report  and  supporting 
documentation  to  the  State  following  the  format  and  time  frames  established  by  the  State.  These 
activities  include: 
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1. 


Evaluating  gathered  information  and  preparing  a  report  of  preliminary  findings; 


2. 


Submitting  reports  of  preliminary  findings  identifying  areas  of  concern  to  the  MCO/PHP; 


3. 


If  the  State  provides  the  MCO/PHP  with  the  opportunity  to  recalculate  performance 
measures  based  on  EQRO  findings,  re-reviewing  selected  performance  measurement 
processes; 


4. 


Evaluating  gathered  information  and  preparation  of  findings  for  the  State;  and 


5. 


Submitting  reports  to  the  State. 


The  protocol  identifies  alternative  approaches  to  determining  the  extent  to  which  the  MCO/PHP 
has  complied  with  requirements  for  calculating  and  reporting  performance  measures.  In  one 
option,  the  EQRO  would  submit  a  summary  of  its  findings  along  with  the  completed  protocol 
assessment  tools  to  the  State  as  supporting  documentation,  but  without  a  validation  designation 
for  individual  performance  measures.  Based  on  the  information  submitted  by  the  EQRO,  the 
State  would  make  a  determination  of  the  extent  to  which  the  MCO/PHP  has  adequately 
calculated  and  reported  the  specified  performance  measures.  Alternatively,  the  EQRO  could 
apply  clearly  defined  decision  rules  established  by  the  State  and  specify  a  validation  finding  for 
each  performance  measure. 
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IV.      PROTOCOL  ACTIVITIES 


PRE-ONSITE  A  CTMTIES 
Objectives  for  Pre-Onsite  Activities: 

The  EQRO  will: 

1 .  understand  the  technical  specifications  for  each  of  the  performance  measures  required  by 
the  State; 

2.  understand  the  State's  requirements  for  performance  measure  reporting  by  the  MCO/PHP 
to  the  State  (e.g.,  report  template,  electronic  submission  format,  etc.);  and 

3.  conduct  and  review  an  assessment  (or  review  the  results  of  a  previously  conducted 
assessment)  of  the  MCO's/PHP's  IS. 

PRE-ONSITE  ACTIVITY  1:        Review  the  State's  requirements  for  MCO/PHP 

performance  measurement  and  reporting. 

The  EQRO  will  need  to  obtain  from  the  State  a  list  of  all  performance  measures  that  the  State 
requires  the  MCO/PHP  to  produce  and  ascertain,  in  consultation  with  the  State,  whether  the 
validation  activities  are  to  include  all  such  measures  or  a  subset  of  those  measures.  The  EQRO 
will  also  need  to  obtain  the  State's  instructions  (specifications)  on  how  the  MCO/PHP  is  to 
calculate  each  performance  measure. 

The  specific  performance  measures  that  a  State  requires  its  Medicaid  MCOs/PHPs  to  report  will 
depend  on  a  number  of  factors  unique  to  each  State.  If  a  State  chooses  to  use  a  set  or  subset  of 
established  standardized  plan-level  performance  measures,  there  are  a  number  of  options  from 
which  to  choose.  These  include  the  NCQA's  HEDIS  measures,  measures  identified  by  the 
Foundation  for  Accountability  (FACCT),  measures  found  in  the  Agency  for  Healthcare  Research 
and  Quality's  CONQUEST  database,  or  measures  suggested  by  MEDSTAT  in  its  publication,  A 
Guide  for  States  to  Assist  in  the  Collection  and  Analysis  of  Medicaid  Managed  Care  Data2.  In 
addition,  States  with  the  resources  and  expertise  to  develop  and  test  the  detailed  specifications 
necessary  for  valid  and  reliable  performance  measures  may  establish  their  own  performance 
measures.  Regardless  of  the  type  or  number  of  performance  measures  chosen  by  the  State,  the 
EQRO  must  understand  the  State's  specifications  (e.g.,  sampling  guidelines,  instructions  for 
calculating  numerators  and  denominators)  for  each  performance  measure,  as  well  as  the  State's 
instructions  to  the  MCO/PHP  for  reporting  the  required  performance  measures  to  the  State. 


^Prepared  under  HCFA  Contract  #500-92-0035.  December  1998. 
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Four  basic  data  collection  methodologies  typically  are  used  to  produce  plan-level  performance 
measures:  1)  use  of  administrative  data,  2)  review  of  medical  records,  3)  use  of  administrative 
data  together  with  medical  record  review  (commonly  called  the  "hybrid"  methodology),  and  4) 
use  of  surveys. 

Use  of  administrative  data  requires  the  MCO/PHP  to  access  data  contained  in  its  management 
information  system(s)  to  calculate  both  the  denominator  and  numerator  of  a  given  performance 
measure.  Such  data  includes  encounter  or  claims  data  (transaction  data)  as  well  as  other 
automated  enrollee  and  provider  information.  The  rate  that  is  reported  is  based  on  information 
found  solely  in  these  administrative  data  sources. 

Calculating  performance  measures  from  medical  record  review  requires  the  visual  inspection  of 
the  medical  records  of  a  sample  of  MCO/PHP  enrollees  to  determine  if  each  enrollee  received  the 
service(s)  in  question  (typically,  this  is  the  numerator  of  the  performance  measure).  Because 
medical  record  reviews  are  time-consuming  and  costly,  most  developers  and  users  of 
performance  measures  are  attempting  to  use,  to  the  extent  feasible,  performance  measures  that 
can  be  calculated  from  administrative  data.  If  medical  record  review  is  unavoidable,  the  less 
costly  and  less  burdensome  "hybrid"  methodology  can  be  used. 

The  hybrid  methodology  combines  the  use  of  administrative  data  with  a  review  of  medical 
records.  The  denominator  of  the  measure  is  first  identified  using  administrative  data  for  a  sample 
of  eligible  members.  The  numerator  is  then  determined  using  data  from  both  administrative  and 
medical  record  reviews.  Typically,  the  MCO/PHP  will  first  query  its  administrative  data  for 
evidence  of  the  numerator  event  for  all  individuals  included  in  the  denominator  sample.  For  any 
member  of  the  sample  who  is  missing  an  administrative  notation  that  the  numerator  service  was 
received,  the  medical  record  is  reviewed. 

Finally,  surveys  also  are  used  to  produce  plan-level  performance  measures.  Surveys  may  include 
information  collected  directly  from  enrollees,  relatives,  primary  caregivers  of  enrollees,  or 
providers  of  healthcare  services.  Administration  and  validation  of  surveys  are  complex  subjects 
and  are  discussed  in  separate  protocols. 

States  may  require  or  allow  MCOs/PHPs  to  report  performance  measures  to  the  State  in  different 
ways.  A  State  may  choose  to  have  plan-level  performance  measures  reported  to  it  in  an  electronic 
format,  such  as  a  comma-delimited,  ASCII  file;  or  it  may  establish  a  set  of  electronic  reporting 
"shells"  that  MCOs/PHPs  fill  out  and  send  to  the  State,  with  attestations  of  the  accuracy  of  the 
information.  States  could  also  allow  hardcopy  submission  of  calculated  performance  measures. 

States  will  also  determine  the  timing  of  the  submission  of  the  calculated  performance  measures. 
Typically,  States  require  performance  measures  to  be  calculated  and  submitted  annually.  The 
annual  submissions  may  be  timed  to  coincide  with  the  end  of  the  State  fiscal  year,  the  calendar 
year,  or  another  reporting  cycle,  such  as  the  reporting  cycle  used  by  NCQA  for  HEDIS 
submissions.  The  EQRO  needs  to  understand  the  expected  dates  and  format  for  MCO/PHP 
reporting. 
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To  facilitate  its  on-site  validation  of  measures,  the  EQRO  should  create  a  "List  of  Performance 
Measures  to  be  Calculated  by  the  MCO/PHP"  (such  as  that  shown  in  TABLE  1)  in  order  to 
understand  the  measures  required  by  the  State,  the  possible  methods  the  MCO/PHP  may  use  to 
collect  them,  and  the  reporting  frequencies  and  format  mandated  by  the  State. 

 TABLE  1 

List  of  Performance  Measures  to  be  Calculated  by  the  MCO/PHP 

(EXAMPLE) 


C  A  1V/TDT  17  IV/f  C  A  C I T  D 17  C 

METHOD  FOR  CALCULATING  PERFORMANCE  MEASURE 

Administrative 
Data 

Medical 

13    AAA   B«  J-l 

tvcCuru 
Review 

Hybrid 

Survey 

Reporting  Frequency  and 

Mr  Kfl  Ilia  I 

The  table  should  have  a 
row  for  each  measure  to 
be  calculated  and  reported 

uy  ilia  Lviy^KJ/ r nr j  uj 

illustrated  below: 

■ 

Childhood  immunization  rate 

Adolescent  immunization  rate 

Percentage  of  enrollees  with 
at  least  one  PCP  visit 

Lead  screening  rate 

Breast  cancer  screeningrate 

Initiation  of  prenatal  care 

Comprehensive  diabetes  care 

Availability  of  language 
interpretation  services 

Follow-up  after 
hospitalization  for  mental 
illnesses 

Women's  chlamydia 
screening  rate 

Rate  of  adverse  asthma  events 
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For  each  measure  in  the  EQRO-created  "List  of  Performance  Measures  to  be  Calculated  by  the 
MCO/PHP,"  the  EQRO  also  should  create  a  separate  performance  measure  validation  worksheet 
that  contains  the  specifications  and  components  of  each  performance  measure  that  is  to  be 
validated,  including:  1)  specifications  for  the  eligible  population  for  the  measure;  2)  data 
collection  methodology;  3)  sampling  methodology  (if  used);  4)  denominator  calculations;  5) 
numerator  calculations;  and  6)  calculated  and  reported  rates.  A  generic  "Performance  Measure 
Validation  Worksheet"  is  found  below  (TABLE  2),  containing  placeholders  for  the  components 
to  be  validated  and  the  elements  to  be  audited.  The  EQRO  should  customized  this  or  a  similar 
worksheet  to  include  the  specifications  (defined  by  the  State)  for  each  performance  measure  to  be 
reported  by  the  MCO/PHP.  For  example,  if  the  measure  is  Breast  Cancer  Screening  (following 
the  HEDIS  specifications),  the  EQRO  would  replace  the  general  "age  and  sex"  categories  in  the 
denominator  portion  of  the  tool  with  the  particular  age  and  sex  specifications  associated  with  that 
measure,  i.e.,  females  between  the  ages  of  52-69.  Using  a  performance  measure  validation 
worksheet  will  improve  the  efficiency  of  the  validation  work  performed  on  site.  An  example  of  a 
completed  Performance  Measure  Validation  Worksheet  is  included  as  ATTACHMENT  I. 

 TABLE  2 

GENERIC  PERFORMANCE  MEASURE  VALIDATION  WORKSHEET 


For  each  performance  measure  to  be  validated  (as  listed  in  TABLE  1  of  this  Pre-Onsite  Activity), 
adapt  the  generic  table  shell  below  to  create  a  validation  worksheet  for  the  ntasure.  [An  example  of 
a  completed  Performance  Measure  Validation  Worksheet  is  included  as  ATTACHMENT  I]. 


PERFORMANCE  MEASURE  { Insert  name  of  performance  measure} 

Validation 
Component 

Audit  Element 

Meets 

Validation 

Requirements 

Yes 

No 

N/A 

Documentation 

Appropriate  and  complete  measurement  plans  and  programming 
specifications  exist  that  include  data  sources,  programming  logic, 
computer  source  code. 

Denominator 

Data  sources  used  to  calculate  the  denominator  (e.g.,  claims  files, 
medical  records,  provider  files,  pharmacy  records)  were  complete  and 
accurate. 

Calculation  of  the  performance  measure  adhered  to  the  specifications  for 
all  components  of  the  denominator  of  the  performance  measure  (e.g., 
member  ID,  age,  sex,  continuous  enrollment  calculation,  clinical  codes 
such  as  ICD-9,  CPT-4,  DSM-IV,  member  months  calculation,  member 
years  calculation,  adherence  to  specified  time  parameters). 

Numerator 

Data  sources  used  to  calculate  the  numerator  (e.g.,  member  ID,  claims 
files,  medical  records,  provider  files,  pharmacy  records,  including  those 
for  members  who  received  the  services  outside  the  MCO/PHP 's  network) 
are  complete  and  accurate. 
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Calculation  of  the  performance  measure  adhered  to  the  specifications  for 
all  components  of  the  numerator  of  the  performance  measure  (e.g., 
clinical  codes  such  as  ICD-9,  CPT-4,  DSM-IV,  pharmacy  data,  relevant 
time  parameters  such  as  admission/discharge  dates  or  treatment  start  and 
stop  dates,  adherence  to  specified  time  parameters,  number  or  type  of 
provider). 

If  medical  record  abstraction  was  used  documentation/tools  were 
adequate. 

If  hvbrid  method  was  used  the  integration  of  administrative  and  medical 
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record  data  was  adequate. 

If  hybrid  method  or  solely  medical  record  review  was  used,  the  results  of 
the  medical  record  review  validation  substantiate  the  reported  numerator. 

bampung 

Sample  was  unbiased. 

Sample  treated  all  measures  independently. 

Sample  size  and  replacement  methodologies  met  specifications. 

Reporting 

State  specifications  for  reporting  performance  measures  were  followed. 

ASSIGNING  A  VALIDATION  FINDING  TO  THE  MEASURE* 


The  validation  finding  for  each  measure  is  determined  by  the  magnitude  of  the  errors  detected  for  the  audit  elements, 
not  by  the  number  of  audit  elements  determined  to  be  "NOT  MET."  Consequently,  it  is  possible  that  an  error  in  a 
single  audit  element  may  result  in  a  designation  of  "NV"  because  the  impact  of  the  error  biased  the  reported 
performance  measure  by  more  than  "x  "  percentage  points.  Conversely,  it  is  also  possible  that  several  audit  element 
errors  may  have  little  impact  on  the  reported  rate  and,  thus,  the  measure  could  be  given  a  designation  of  "SC."  The 
following  is  a  list  of  validation  findings  and  their  corresponding  definitions: 

FC        =  Fully  Compliant 

Measure  was  fully  compliant  with  State  specifications. 

SC        =  Substantially  Compliant 

Measure  was  substantially  compliant  with  State  specifications  and  had  only  minor  deviations  that 
did  not  significantly  bias  the  reported  rate. 

NV       =  Not  Valid 

Measure  deviated  from  State  specifications  such  that  the  reported  rate  was  significantly  biased. 
This  designation  is  also  assigned  to  measures  for  which  no  rate  was  reported,  although  reporting  of 
the  rate  was  required. 

NA        =  Not  Applicable 

Measure  was  not  reported  because  MCO/PHP  did  not  have  any  Medicaid  enrollees  that  qualified 
for  the  denominator. 


AUDIT  DESIGNATION 


*  Assigning  a  validation  finding  to  a  measure  is  discussed  in  Post-Onsite  Activity  1.  This  material  is  included 
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here  because  it  should  be  part  of  a  performance  measure  validation 
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PRE-ONSITE  ACTIVITY  2:        Prepare  the  MCO/PHP  for  EQRO  On-site  Activities. 


Prior  to  conducting  onsite  activities,  the  EQRO  will  contact  the  MCO/PHP  in  order  to: 

•  explain  the  procedures  and  time  line  for  performance  measure  validation  activities; 

•  request  identification  of  personnel  within  the  MCO/PHP  who  will  be  responsible  for 
responding  to  EQRO  requests  for  documentation  or  information,  as  well  as  scheduling 
activities  and  interviews;  and 

•  communicate  the  EQRO's  policies  and  procedures  with  respect  to  safeguarding 
confidential  information. 

An  introductory  letter  to  the  MCO/PHP  should  discuss  the  above  issues  and  explain  the  EQRO's 
potential  need  to  interview  MCO/PHP  personnel,  so  that  interviewees  are  prepared  in  terms  of 
time  and  information.  Potential  interviewees  include  any  MCO/PHP  or  vendor  staff  whose  areas 
of  expertise  or  responsibility  relate  to  performance  measurement  and  whose  insights  might 
improve  the  EQRO's  understanding  of  MCO/PHP  processes  to  calculate  or  report  performance 
measures.  These  include,  for  example:  the  Director  of  Health/Medical  Information  Systems,  IS 
programmers  or  operators,  Director  of  Member/Patient  Services,  Director  of  Utilization 
Management,  and  the  Director  of  Quality  Improvement. 

Also,  the  EQRO  will  provide  to,  or  request  from,  the  MCO/PHP  four  other  types  of  information 
in  preparation  for  its  on-site  activities: 

1 .  a  list  and  description  of  all  State-required  performance  measures  calculated  by  or  on 
behalf  of  the  MCO/PHP; 

2.  a  list  of  all  enrollees  (or  enrollee  identifiers)  included  in  the  numerators  of  performance 
measures  calculated  wholly  or  in  part  by  medical  record  review; 

3.  a  list  of  documents  that  the  EQRO  may  potentially  review  during  onsite  activities;  and 

4.  background  information  on  the  MCO's/PHP's  Information  Systems. 

1.  List  of  performance  measures  calculated  by  the  MCO/PHP.  This  list  of  performance 
measures  calculated  by  the  MCO/PHP  (TABLE  3)  is  similar  to  the  list  completed  by  the  EQRO 
during  Pre-Onsite  Activity  1  (TABLE  1).  However,  while  the  TABLE  1  list  was  prepared  by  the 
EQRO  to  familiarize  itself  with  the  State's  requirements  for  performance  measures,  TABLE  3  is 
sent  to  the  MCO/PHP  by  the  EQRO  for  the  MCO/PHP  to  complete.  The  MCO/PHP  is  to  insert 
into  the  table,  next  to  each  performance  measure  listed  in  the  table,  information  on  the  methods 
the  MCO/PHP  used  to  calculate  the  performance  measures  required  by  the  State.  This  is 
especially  important  for  those  measures  for  which  the  MCO/PHP  has  a  choice  of  methods  to  use 
for  their  calculation;  e.g.,  administrative,  medical  record  review,  or  hybrid  data  collection 
methodologies.  The  EQRO  should  send  to  the  MCO/PHP  the  same  list  of  measures  contained  in 
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TABLE  1,  but  with  a  modified  title  and  instructions  (as  illustrated  in  TABLE  3,  below)  to  reflect 
that  the  MCO/PHP  is  to  complete  the  table  and  return  it  to  the  EQRO. 

TABLE  3 

List  of  Performance  Measures  Calculated  by  the  MCO/PHP  -  Example 

Instructions  to  MCOs/PHPs:  For  each  measure  the  State  requires  you  to  report  (in  column  I), 
indicate  the  method(s)  your  MCO/PHP  used  to  produce  it  by  checking  columns  2  -  5,  as 
appropriate.  In  column  6  note  the  reporting  frequencies  (e.g.,  quarterly,  annually)  and  format 
(e.g.,  paper  report,  electronic  medium)  your  MCO/PHP  has  used  (or  expects  to  use)  to  report  to 
the  State.  Return  this  table  to  (name  of  EQRO)  by  (date),  so  that  this  information  may  be 
reviewed  prior  to  our  site  visit  to  validate  your  MCO  's/PHP 's  performance  measures. 


Measure 
{Examples} 

Administrative 

Medical 
Record 
Review 

Hybrid 

Survey 

Reporting 
Frequency 
and  Format 

The  table  should  contain  a 
row  jor  each  measure  to  be 
calculated  and  reported  by 
the  MCO/PHP. 

Childhood  immunization  rate 

Adolescent  immunization  rate 

Percent  of  enrollees  with  at 

IcdSl  One  rLr  VISU 

Lead  screening  rate 

Breast  cancer  screening  rate 

Initiation  of  prenatal  care 

Comprehensive  diabetes  care 

Language  interpretation 
services  -  availability 

Follow-up  after 
hospitalization  for  mental 
illnesses 

Women's  chlamydia 
screening  rate 

Rate  of  adverse  asthma  events 
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2.  A  list  of  all  enrollees  (or  enrollee  identifiers)  included  in  the  numerators  of  all  measures 
calculated  in  part  or  wholly  from  medical  record  review.  For  each  of  at  least  three 
performance  measures  which  the  MCO/PHP  calculated  either  entirely  by  medical  record  review 
or  by  the  hybrid  methodology,  the  EQRO  will  review,  onsite,  30  medical  records  found  to  meet 
numerator  requirements.  The  purpose  of  this  review  is  to  verify  the  accuracy  of  the  medical 
record  review  conducted  by  each  MCO  PHP. 

To  provide  sufficient  time  for  each  MCO/PHP  to  gather  the  required  medical  record 
documentation,  the  MCO/PHP  will  need  to  identify  to  the  EQRO,  prior  to  the  EQRO's  on-site 
visits:  1)  all  performance  measures  calculated  through  medical  record  review  or  the  hybrid 
methodology  (obtained  by  completing  TABLE  3),  and  2)  for  measures  which  used  medical  • 
record  review  or  the  hybrid  methodology  and  selected  by  the  EQRO,  a  list  of  enrollees  included 
in  the  numerator  for  each  measure  as  a  result  of  positive  findings  through  medical  record  review. 
From  this  list,  the  EQRO  will  select  30  members  for  each  performance  measure.  The  MCO/PHP 
will  then  be  asked  to  make  available  the  medical  records  or  copies  of  medical  records  for  these 
enrollees  at  the  time  of  the  on-site  visit.  In  cases  where  there  are  fewer  than  30  numerator 
positives,  the  EQRO  will  review  all  records  for  that  measure. 

3.  List  of  potential  validation  documents  and  processes.  The  List  of  Potential  Validation 
Documents  and  Processes  (ATTACHMENT  II)  identifies  documents  and  information 
concerning  the  MCO's  PHP's  data  sources  and  processes  that  the  EQRO  may  review  during  the 
course  of  the  validation  activities.  This  list  is  intended  to  assist  the  MCO  PHP  in  preparing  for 
the  validation  audit. 

4.  Information  Systems  Capabilities  Assessment  Tool  (ISCA).  The  EQRO  will  send  an 
Information  Systems  Capabilities  Assessment  Tool  (ISCA)  to  the  MCO/PHP.  to  be  completed 
and  returned  to  the  EQRO  prior  to  the  on-site  visit.  The  ISCA  consists  of  questions  and  requested 
documentation  to  provide  the  EQRO  with  background  information  on  the  MCO's/PHP's 
policies,  processes,  and  data  needed  for  the  onsite  validation  activities.  The  ISCA  is  discussed  in 
detail,  in  Pre-Onsite  Activity  3. 
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PRE-ONSITE  ACTIVITY  3:         Assess  the  integrity  of  the  MCO'S/PHP's  information 

system. 

Complete  and  accurate  data  is  key  to  valid  and  reliable  performance  measurement.  If  these  two 
data  characteristics  are  not  maintained,  then  calculated  measures  become  flawed,  and  their 
validity  jeopardized.  Therefore,  prior  to  validating  individual  performance  measures,  the  EQRO 
must  first  assess  the  integrity  of  the  MCO's/PHP's  IS  and  the  completeness  and  accuracy  of  the 
data  produced  by  that  system. 

Methods  of  Evaluation 

Prior  to  conducting  the  onsite  visit,  the  EQRO  should  send  to  the  MCO/PHP  an  Information 
Systems  Capabilities  Assessment  Tool  (ISCA)  such  as  that  located  in  Appendix  A.  The  ISCA 
asks  questions  of  and  requests  documentation  from  the  MCO/PHP  in  order  to  provide 
information  on  the  MCO's/PHP's  IS  policies  and  procedures  to  help  focus  onsite  validation 
activities.  The  ISCA  found  in  Appendix  A  corresponds  to  the  key  objectives  identified  in  this 
protocol.  The  first  section  of  the  ISCA  provides  general  background  information  on  the 
MCO/PHP.  Subsequent  sections  address  the  structural  components  of  the  IS,  focusing  on  the 
collection  of  administrative,  encounter,  and  clinical  data,  and  the  consolidation  or  coordination 
of  those  data  files  for  use  in  performance  measurement  and  quality  improvement  activities. 

The  ISCA  also  requests  information  from  the  MCO/PHP  concerning  the  conduct  and  timing  of 
any  other  recent,  independent,  documented  assessment  of  its  IS.  An  assessment  may  already  have 
been  conducted  by  the  State  itself  or  by  another  entity.  IS  assessment  could  have  been  performed 
as  a  component  of  validating  encounter  data  or  determining  compliance  with  Medicaid  standards 
pertaining  to  MCO/PHP  ISs.  If  the  MCO/PHP  has  not  had  an  IS  capability  assessment 
completed,  or  has  not  had  one  completed  within  a  time  frame  that  meets  State  specifications3,  the 
EQRO  will  conduct  an  IS  assessment  as  part  of  this  protocol,  using  an  information  systems 
assessment  tool,  such  as  that  in  Appendix  A.  Alternatively,  if  the  MCO/PHP  recently  had  an 
independent  assessment  of  its  IS,  the  EQRO  could  review  the  results  of  this  prior  assessment. 

The  EQRO  should  assess  the  MCO's/PHP's  IS  using  questions  and  approaches  such  as  those 
contained  in  Appendix  A,  or  review  the  results  of  a  recent  IS  assessment  consistent  with  the 
content  in  Appendix  A.  This  will  ensure  that  auditors  are  familiar  with  the  strengths  and 


Each  State  will  determine  the  frequency  with  which  it  wants  an  MCO's/PHP's  IS  capability  assessment  to 
take  place  (thereby  determining  the  length  of  time  such  an  assessment  is  valid).  On  the  one  hand,  the  process  is  time- 
and  resource-intensive,  so  limiting  the  burden  on  the  MCO/PHP  should  be  a  factor  in  the  determination.  On  the  other 
hand,  IS  technology  changes  rapidly,  so  the  State  should  ensure  that  changes  to  an  MCO/PHP's  IS  are  assessed 
frequently  enough  to  ensure  that  the  structure  and  function  continue  to  be  adequate  for  the  State-required  tasks. 
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weaknesses  of  the  MCO's/PHP's  IS.  As  the  EQRO  reviews  the  IS  assessment  report,  it  should 
pay  close  attention  to  the  strengths  and  weaknesses  of  the  MCO's/PHP's  IS  with  respect  to  the 
types  of  data  frequently  used  in  plan-level  performance  measures,  such  as  data  on: 
membership/enrollment,  claims/encounters,  providers,  laboratory  and  pharmacy  services,  and 
medical  record  data.  Some  of  the  characteristics  commonly  associated  with  these  data  elements 
that  may  affect  performance  measures  are: 


Membership/Enrollment  Data.  Elements  of  the  membership  or  enrollment  database  will 
vary  by  MCO/PHP.  However,  for  the  purposes  of  plan-level  performance  measurement, 
the  membership  or  enrollment  database  should  capture  at  least  the  following  information: 

age/date  of  birth. 

enrollment  and/or  termination  dates.  (Note:  The  MCO's/PHP's  data 
system  should  be  able  to  track  multiple  enrollment  and  termination  dates). 

primary  care  provider. 

member  identification  number  such  as  the  member's  social  security 
number,  MCO-  or  PHP-designated  number,  State-issued  Medicaid 
number,  HCFA-issued  Medicare  number.  (Note:  Be  aware  of  cases  in 
which  more  than  one  member  may  exist  under  the  same  identification 
number  within  the  system;  or  in  which  the  same  member  may  exist  under 
more  than  one  identification  number  within  the  system;  or  in  which  a 
member's  identification  number  may  change  through  re-enrollment,  name 
change,  or  switch  in  product-line  coverage). 

The  EQRO  also  should  be  aware  of  whether  the  MCO/PHP  has  processes  in  place  to 
periodically  ensure  that  enrollment/membership  data  are  current  and  accurate,  particularly 
at  the  time  it  runs  its  source  code/computer  programs  to  identify  denominators  for  plan- 
level  performance  measures. 

Further,  the  EQRO  should  be  aware  of  changes  in  the  MCO's/PHP's  membership  data 
systems  that  might  affect  the  production  of  the  plan-level  performance  measures.  Major 
changes,  upgrades  or  consolidations  within  the  system,  or  acquisitions/mergers  with  other 
MCOs/PHPs  may  impact  the  accuracy  or  completeness  of  any  of  the  data  elements, 
which,  in  turn,  may  impact  the  validity  of  the  reported  measures. 

Provider  Data.  Elements  of  the  provider  data  set  should  typically  include: 

Designation  as  a  primary  care  physician  and/or  providers'  specialty. 

Provider  identification  number,  such  as  a  Tax  ID  number,  or  MCO-  or 
PHP-designated  number.  (Note:  Though  it  may  be  less  common  to  see 
duplication  of  provider  numbers  within  a  provider  database  than 
duplication  of  member  identifications  within  a  membership/enrollment 
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database,  the  EQRO  should  be  aware  of  any  circumstances  in  which  more 
than  one  provider  can  exist  with  the  same  identification  number  within  the 
system,  or  circumstances  in  which  the  same  provider  may  have  more  than 
one  identification  number  within  the  system). 

Providers  with  more  than  one  office  location. 

Providers  with  closed  panels  (i.e.,  provider  availability). 

Provider  start  and  termination  dates. 

Provider  certification  data  such  as  licensure,  provider 
residency/fellowship,  date  and  specialty  of  Board  Certification  status. 


The  EQRO  should  be  aware  of  whether  the  MCO/PHP  has  processes  in  place  to 
periodically  ensure  that  provider  data  are  current  and  accurate  for  all  types  of  providers 
(individual  providers,  provider  groups,  provider  networks,  contracted  vendors).  This 
becomes  particularly  important  at  the  time  the  MCO/PHP  runs  its  source  code/computer 
programs  to  identify  elements  of  plan-level  performance  measures. 


Further,  the  EQRO  should  be  aware  of  changes  in  the  MCO's/PHP's  provider  data 
systems  that  might  affect  the  production  of  the  performance  measures.  Major  changes, 
upgrades  or  consolidations  within  the  system,  or  acquisitions/mergers  with  other 
MCOs/PHPs  may  impact  the  accuracy  or  completeness  of  any  of  the  data  elements, 
which,  in  turn,  may  impact  the  validity  of  the  reported  measures. 


•        Claims  Data  and  Encounter  Data.  Claim/encounter  data  should  cover  all  types  of 
services  offered  by  the  MCO/PHP,  such  as:  behavioral  health,  family  planning,  home 
health  care,  hospital,  laboratory,  pharmacy,  primary  care,  radiology,  specialty  care,  vision 
care.  These  data  typically  include  the  following  elements: 


-  Patient  ID 
-Sex 

-  Date  of  birth 

-  Last  date  of  service 

-  Primary  diagnosis 

-  Primary  procedure 

-  Revenue  codes 

-  Provider  specialty 


-  Name 
-Age 

-  First  date  of  service 

-  Provider  specialty 

-  Secondary  diagnosis 

-  Secondary  procedure 

-  Provider  ID 

-  Discharge  status 


For  each  type  of  claim/encounter  data  captured,  the  EQRO  should  be  aware  of:  1)  the 
total  number  of  diagnosis  and  procedure  codes  that  can  be  captured  by  the  system;  2) 
whether  or  not  principal  or  secondary  diagnosis  or  procedure  codes  can  be  accurately 
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distinguished  in  the  system;  and  3)  the  maximum  number  of  digits  or  characters  the 
system  captures  for  each  type  of  claim/encounter.  For  many  plan-level  performance 
measures,  the  accuracy  and  validity  of  the  measure  may  be  adversely  affected  if  the 
MCO's/PHP's  IS  is  unable  to  collect  and/or  differentiate  among  a  sufficient  number  of 
codes. 

The  various  coding  systems  and  forms  used  by  the  MCO/PHP  and  its  vendors  to  capture 
clinical  information  through  its  claims  and  encounter  databases  are  relevant  to  validating 
plan-level  performance  systems.  Coding  systems  are  formal,  standardized  approaches 
(such  as  ICD-9,  CPT-4,  DSM-IV,  revenue  codes,  or  internally  developed  codes)  to 
categorize  types  of  encounters  and  procedures  by  data  elements  such  as  inpatient  and 
ambulatory  diagnoses  and  procedures  for  medical,  surgical,  or  mental  health/substance 
abuse  encounters/claims.  Note  that  internally-developed  codes  may  be  particularly 
problematic.  The  EQRO  should  understand  how  the  MCO's/PHP's  IS  translates  or  maps 
these  codes  back  to  standard  codes  for  plan-level  performance  measure  reporting,  and 
how  it  ensures  the  accuracy  of  these  translation  processes. 

Medical  Record  Data.  In  cases  where  medical  records  are  accessed  to  obtain  information 
for  calculating  plan-level  performance  measures,  the  EQRO  should  be  aware  of  how  the 
MCO/PHP  retrieves  information  from  medical  records.  For  example,  the  training  and 
tools  that  medical  record  review  staff  receive  may  affect  the  accuracy  and  completeness 
of  the  data  retrieval.  A  second  area  of  concern  is  how  medical  record  data  is  entered  into 
any  database  that  will  be  used  to  produce  the  performance  measures. 

Pharmacy  and  Laboratory  Data.  A  key  issue  commonly  encountered  with  pharmacy  or 
laboratory  data  for  Medicaid  managed  care  plans  is  that  these  services  are  frequently 
contracted  out  to  a  variety  of  providers.  Ideally,  pharmacy  data  will  use  standardized 
codes  for  prescription  drugs  such  as  those  promulgated  by  the  National  Council  for 
Prescription  Drug  Programs  (NCPDP),  and  laboratory  services  will  use  a  similar, 
nationally  recognized  system  of  coding.  However,  the  diverse  nature  of  the  size,  type,  and 
ownership  of  pharmacy  and  laboratory  providers  should  lead  the  EQRO  to  anticipate 
wide  variations  in  the  use  of  standardized  coding  and  a  multitude  of  unique  "home 
grown"  codes.  These  non-standard  coding  schemes  require  that  the  MCO/PHP  have  a 
system  to  develop  crosswalks  among  these  different  codes  in  order  to  store  the  necessary 
information  in  its  performance  measure  database.  As  with  the  assessment  of  the 
claims/encounter  data  systems,  the  EQRO  should  understand  not  only  the  MCO's/PHP's 
system  of  mapping  non-standard  pharmacy  and  lab  codes  to  standardized  codes,  but  the 
mechanism  the  MCO/PHP  uses  to  ensure  the  accuracy  of  these  translation  processes. 

If  pharmacy  or  laboratory  data  are  not  collected  through  an  administrative  or  claims 
database,  pharmacy  or  lab  data  may  be  present  in  medical  records.  However,  relying  on 
medical  records  to  supply  pharmacy  or  laboratory  data  is  problematic  because  of 
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obstacles  such  as  non-standard  coding  and  terminology  and  poor  coordination  of  records 
and  record  linkages  between  primary  care  and  specialist  providers.  The  EQRO  should  be 
aware  of  these  issues  and  question  providers  on  the  reliability  of  medical  record  data  and 
pharmacy  data  as  appropriate. 

In  addition,  for  many  plan-level  performance  measures,  the  IS  will  need  to  be  able  to  link  these 
different  sources  of  data.  For  example,  in  order  to  identify  enrollees  with  diabetes,  an  MCO/PHP 
may  have  to  combine  diagnosis  code  data  from  inpatient  or  ambulatory  encounters  (not  all 
ongoing  conditions  are  reported  at  every  encounter)  with  pharmacy  data,  lab  data,  and/or  a 
disease  registry  if  one  exists.  To  determine  whether  these  diabetic  enrollees  have  received  a 
retinal  examination  from  an  ophthalmologist  or  optometrist  within  the  previous  year,  the 
MCO/PHP  would  have  to  link  procedure  code  data  from  either  encounter  forms,  medical  records, 
or  claims  with  information  about  the  specialty  of  the  providers  that  performed  the  examinations 
for  these  members. 

The  EQRO  will  analyze  the  results  of  the  assessment  of  the  MCO's/PHP's  IS  and  determine  the 
implications  of  the  findings  for  the  ability  of  the  MCO/PHP  to  calculate  the  performance 
measures  specified  by  the  State.  The  EQRO  will  evaluate  MCO/PHP  answers  against  IS 
capabilities  necessary  to  accurately  and  completely  calculate  and  report  the  specific  plan-level 
performance  measures  mandated  by  the  State,  and  will  identify  any  problem  areas  or  items  in 
need  of  clarification.  Where  an  answer  seems  incomplete,  or  indicates  an  inadequate  process,  the 
EQRO  notes  this  issue  for  follow-up  and  further  review  during  the  onsite  activities.  This  will 
help  the  onsite  validation  activities  focus  on  the  areas  most  likely  to  be  an  issue  in  the  validation 
process.  In  addition,  knowledge  gained  from  the  ISCA  provides  a  knowledge  base  for  effective 
interviews  with  key  MCO/PHP  staff. 
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ONSITE  A  CTIVITIES 


Objectives  for  Onsite  Activities: 

The  EQRO  will  evaluate  the  extent  to  which  the  MCO/PHP  has: 

1 .  adequate  data  integration  and  control  procedures  for  accurate  production  of  the  State- 
specified  performance  measures; 

2.  complete  and  accurate  documentation  of  data  and  processes  used  to  calculate  and  report 
the  State-specified  performance  measures;  and 

3.  correctly  implemented  appropriate  processes  for  calculating  and  reporting  the  State- 
specified  performance  measures. 


ONSITE  ACTIVITY  1:       Assess  data  integration  and  control. 

In  the  last  activity  (Pre-Onsite  Activity  3),  the  EQRO  examined  background  information  on  the 
capability  of  the  MCO's/PHP's  IS  to  collect  and  integrate  valid  data  from  sources  internal  and 
external  to  the  MCO/PHP.  This  on-site  activity  further  assesses:  1 )  the  MCO's/PHP's  ability  to 
link  data  from  multiple  sources  in  order  to  calculate  the  State-mandated  performance  measures; 
and  2)  whether  the  MCO/PHP  has  used  these  abilities  in  a  manner  that  ensures  the  accuracy  of 
the  calculated  performance  measures.  This  assessment  will  be  accomplished  through: 

1 .  Review  of  documentation,  procedures,  and  data  pertaining  to  the  MCO's/PHP's  IS,  and 

2.  Interviews  of  MCO/PHP  personnel  with  knowledge  of  the  MCO's/PHP's  IS  and  its 
application  to  performance  measurement. 

ATTACHMENT  III,  IS  Data  Integration  and  Control  -  Documentation  Review  Worksheet  lists 
documents,  data,  and  procedures  to  be  examined  to  assess  MCO/PHP  data  integration  and 
control.  EQROs  should  use  a  worksheet  such  as  ATTACHMENT  III  to  document  their  findings. 
In  examining  the  MCO's/PHP's  documentation,  procedures  and  data,  the  EQRO  should: 

1 .  Examine  for  accuracy  and  completeness  the  details  of  the  MCO's/PHP's  processes  to 
transfer  data  from  membership,  provider,  encounter/claims,  and  other  data  files  into  a 
data  repository  (or  use  of  other  mechanism(s)  to  consolidate  data)  to  calculate 
performance  measures  and  to  keep  the  data  until  the  calculations  of  the  performance 
measures  have  been  completed  and  validated. 

2.  Examine  samples  of  data  from  the  data  repository  and  transaction  files  to  assess 
completeness  and  accuracy. 
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3.  Investigate  the  MCO's/PHP's  processes  to  consolidate  diversified  files  and  extract 
required  information  from  a  performance  measure  repository  or  other  data  consolidation 
file. 

4.  Compare  actual  results  of  file  consolidations  or  extracts  to  those  which  should  have 
resulted  according  to  documented  algorithms  or  specifications. 

5.  Review  procedures  for  coordinating  the  activities  of  multiple  subcontractors  to  ensure 
accurate,  timely,  and  complete  integration  of  the  data  into  the  performance  measure 
database. 

6.  Review  computer  program  reports  or  documentation  that  reflect  these  vendor 
coordination  activities  and  spot  check  to  verify  that  no  data  necessary  to  performance 
measure  reporting  are  lost  or  inappropriately  modified  during  transfer. 

7.  If  the  MCO/PHP  uses  a  data  repository  (or  data  warehouse),  evaluate  its  structure  and 
format  and  examine  program  flow  charts  and  source  codes  to  determine  the  extent  to 
which  the  repository/warehouse  enables  and  has  enabled  analyses  and  reports. 

8.  Assess  the  extent  to  which  proper  linkage  mechanisms  have  been  employed  to  join  data 
from  all  necessary  sources  (e.g.,  identifying  a  member  with  a  given  disease/condition). 

9.  Examine  and  assess  the  adequacy  of  the  documentation  governing  the  performance 
measures  production  process,  including  MCO/PHP  production  activity  logs,  and 
MCO/PHP  staff  review  of  report  runs. 

10.  Review  documentation  that  confirms  that  prescribed  data  cutoff  dates  were  followed. 

11.  If  appropriate,  request  that  the  MCO/PHP  demonstrate  it  has  retained  copies  of  files  or 
databases  used  for  performance  measure  reporting,  in  the  event  that  results  need  to  be 
reproduced. 

12.  Review  documentation  standards  that  assure  that  the  performance  measure  reporting 
software  program  is  properly  documented  with  respect  to  every  aspect  of  the  reporting 
repository,  including  building,  maintaining,  managing,  testing,  and  report  production. 

13.  Review  the  MCO's/PHP's  process  and  documentation  to  ensure  that  it  complies  with  the 
MCO/PHP  standards  associated  with  the  performance  measure  reporting  program 
specifications,  code  review,  and  testing. 

In  addition,  as  needed,  the  EQRO  should  supplement  the  direct  examination  of  IS  policies, 
procedures,  and  data  with  interviews  of  MCO/PHP  personnel.  MCO/PHP  personnel  who  can 
potentially  provide  helpful  information  include:  the  Director  of  Health/Medical  Information 
Systems,  system  programmers  or  operators,  and  selected  sub-contractors.  An  Interview  Guide 
and  suggested  questions  to  ask  during  these  interviews  are  located  at  ATTACHMENT  IV,  Guide 
for  Interviews  of  MCO/PHP  Personnel  Concerning  Data  Integration  and  Control. 

The  EQRO  should  document  all  findings  with  respect  to  the  adequacy  of  the  MCO's/PHP's  data 
integration  and  control  procedures  on  a  worksheet  such  as  that  found  in  ATTACHMENT  V, 
Data  Integration  and  Control  Findings  -  Documentation  Worksheet. 
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ONSITE  ACTIVITY  2:       Assess  documentation  of  data  and  processes  used  to  calculate 

and  report  performance  measures. 


The  MCO/PHP  should  have  documentation  of  all  steps  undertaken  in  the  production  of  the  State- 
specified  performance  measures,  including  documentation  of:  1)  the  collection  of  data  from 
various  sources  (e.g.,  membership,  enrollment,  provider,  claims,  or  encounter  files;  medical 
records;  laboratory  or  pharmacy  records);  2)  steps  taken  to  integrate  the  required  data  into  a 
performance  measure  data  set  or  repository;  and  3)  procedures  or  programs  to  query  the  data 
set/repository  to  identify  denominators,  generate  appropriate  samples,  determine  numerators,  and 
apply  proper  algorithms  to  the  data  in  order  to  produce  valid  and  reliable  performance  measures. 


During  this  activity,ybr  each  measure  to  be  validated,  the  EQRO  will: 


1 .        Review  performance  measurement  plans  and  policies  to  assess  the  extent  to  which  they 
include: 


data  file  and  field  definitions; 

maps  to  standard  coding  if  not  used  in  original  data  collection;  and 

statistical  testing  of  results,  and  any  corrections  or  adjustments  made  after 
processing. 


2.        Examine  documentation  (which  may  be  either  a  schematic  diagram  or  in  narrative  form) 
of  programming  specifications  to  ensure  that  documentation  exists  for  at  least  the 
following  information: 


a  project  or  measurement  plan,  including  work  flow. 

all  data  sources;  including  external  data  (whether  from  a  vendor,  public  registry, 
or  other  outside  source)  and  any  prior  years'  data  (if  applicable). 

documentation  of  the  original  universe  of  data  that  includes  record-level  patient 
identifiers  that  can  be  used  to  validate  entire  programming  logic  for  creating 
denominators,  numerators,  and  samples. 

detailed  medical  record  review  methods  and  practices,  including  the  qualifications 
of  medical  record  review  supervisor  and  staff;  reviewer  training  materials;  audit 
tools  used,  including  completed  copies  of  each  record-level  reviewer 
determination;  all  case-level  critical  performance  measure  data  elements  used  to 
determine  a  positive  or  negative  event  or  exclude  a  case  from  same;  and  inter-rater 
reliability  testing  procedures  and  results. 

detailed  computer  queries,  programming  logic,  or  source  codes  used  to  create  all 
denominators,  numerators,  and  samples  (if  applicable  to  the  measure).  This 
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includes  the  processes  for  identifying  the  population  or  sample  for  the 
denominator  and/or  numerator  for  each  measure.  If  sampling  is  used,  this  includes 
a  description  of  sampling  techniques  and  documentation  that  samples  used  for 
baseline  and  repeat  performance  measurements  were  chosen  using  the  same 
sampling  frame  and  methodology. 

•         documentation  of  calculation  for  changes  in  performance  from  previous  periods 
(if  applicable)  including  statistical  tests  of  significance. 

The  EQRO  will  need  to  refer  to  the  specifications  for  each  measure  that  were  developed  by  the 
EQRO  during  Pre-Onsite  activities  (illustrated  in  ATTACHMENT  I).  A  list  of  the 
documentation  to  be  reviewed  is  located  at  ATTACHMENT  VI,  Data  and  Processes  Used  to 
Calculate  and  Report  Performance  Measures  -  Documentation  Review  Worksheet.  In  addition,  as 
needed,  the  EQRO  will  interview  the  Director  of  Health/Medical  Information  Systems,  system 
programmers  or  operators,  and  the  Director  of  Quality  Improvement  or  other  MCO/PHP 
personnel  to  supplement  this  information,  facilitate  demonstrations  of  performance  measurement 
processes,  and  provide  the  answers  to  questions  such  as  the  following: 

1 .  How  are  policies  governing  documentation  of  data  requirements  for  performance 
measurement,  (e.g.,  data  file  and  field  definitions,  mapping  between  standard  and  non- 
standard codes)  updated  and  enforced?  Who  is  responsible  for  this? 

2.  How  are  programming  specifications  for  plan  level  performance  measures  documented? 
Who  is  responsible  for  this? 

3.  Are  the  documentation  processes  up  to  date? 

The  results  of  the  EQRO's  review  of  the  MCO's/PHP's  documentation  of  data  and  processes 
used  to  prepare  and  submit  performance  measures  should  be  recorded  on  a  form  such  as  that 
found  as  ATTACHMENT  VII:  Data  and  Processes  Used  to  Calculate  and  Report  Performance 
Measures  -  Documentation  Worksheet. 
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ONSITE  ACTIVITY  3:       Assess  processes  used  to  produce  denominators. 

The  fundamental  question  to  be  answered  by  validating  the  calculation  of  the  denominator(s)  of 
performance  measures  is  to  what  extent  the  MCO/PHP  used  the  appropriate  data  (including 
linked  data  from  separate  data  sets)  to  identify  the  entire  at-risk  population.  The  "appropriate 
data"  will  vary  from  measure  to  measure,  depending  on  criteria  such  as  age,  sex,  diagnosis,  or 
procedure,  and  may  be  adjusted  to  exclude  certain  patients  for  reasons  identified  in  the 
specifications  established  by  the  State  for  calculating  the  measure.  Also,  in  some  cases,  the 
MCO  PHP  may  have  to  estimate  portions  of  the  population,  such  as  newborns,  who  cannot 
always  be  readily  and  fully  counted.  In  such  cases,  the  EQRO  should  confirm  that  the 
methodology  used  for  such  estimations  is  valid.  In  conducting  this  activity,  the  EQRO  will  need 
to  refer  to  the  State's  specifications  for  each  measure  as  noted  by  the  EQRO  during  Pre-Onsite 
activities  and  as  illustrated  in  ATTACHMENT  I. 

During  this  activity, ybr  each  performance  measure  calculated  by  the  MCO/PHP  and  chosen  to 
be  included  in  the  validation  activity,  the  EQRO  will  assess  the  extent  to  which: 

1 .  all  members  who  were  eligible  to  receive  the  specified  services  under  study  were 
included  in  the  initial  population  from  which  the  final  denominator  was  produced.  This 
"at  risk"  population  will  include  both  members  who  received  the  services,  as  well  as 
those  who  did  not.  This  same  validation  activity  applies  to  provider  groups,  or  other 
relevant  populations  identified  in  the  specifications  of  each  performance  measure. 

2.  programming  logic  or  source  codes  which  identify,  track,  and  link  member  enrollment 
within  and  across  product  lines  (e.g.,  Medicare  and  Medicaid),  by  age  and  gender,  as  well 
as  through  possible  periods  of  enrollment  and  disenrollment,  have  been  appropriately 
applied  according  to  the  specifications  of  each  performance  measure.  This  is  determined 
by  evaluating  the  extent  to  which: 

•  calculations  of  continuous  enrollment  criteria  were  correctly  carried  out  and 
applied  to  each  measure  (if  applicable). 

the  MCO/PHP  used  appropriate  mathematical  operations  to  determine  patient  age 
or  range. 

•  the  MCO/PHP  can  identify  the  variable(s)  that  code  the  member's  sex  in  every 
file  or  algorithm,  and  that  the  MCO  PHP  can  explain  what  classification  is  carried 
out  if  neither  of  the  required  codes  is  present. 

3.  the  MCO  PHP  has  correctly  calculated  member  months  and  member  years,  if  applicable 
to  the  performance  measure. 

4.  the  MCO  PHP  has  properly  evaluated  the  completeness  and  accuracy  of  any  codes  used 
to  identify  medical  events,  such  as  diagnoses,  procedures,  or  prescriptions,  and  that  these 
codes  have  been  appropriately  identified  and  applied  as  specified  in  each  performance 
measure. 
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5.  time  parameters  required  by  the  performance  measure  specifications  are  followed  (e.g., 
cut-off  dates  for  data  collection,  counting  30  calendar  days  after  discharge  from  a 
hospital). 

6.  performance  measure  specifications  or  definitions  were  followed  in  excluding  members 
from  a  denominator.  For  example,  if  a  measure  relates  to  receipt  of  a  specific  service,  the 
denominator  may  need  to  be  adjusted  to  reflect  instances  in  which  the  patient  refuses  the 
service  or  the  service  is  contraindicated. 

7.  systems  or  methods  used  by  the  MCO/PHP  to  estimate  populations  when  they  cannot  be 
accurately  or  completely  counted  (e.g.,  newborns)  are  valid. 

Policies,  procedures,  data,  and  information  to  be  reviewed  in  conducting  these  activities  are  listed 
in  ATTACHMENT  VIII.  Information  obtained  from  a  review  of  these  policies,  procedures,  data, 
and  information  should  be  supplemented  and  confirmed,  as  needed,  though  interviews  with 
MCO/PHP  personnel,  including:  the  Director  of  Health/Medical  Information  Systems,  system 
programmers  or  operators,  and  selected  sub-contractors.  Suggested  questions  to  be  asked  are 
located  in  ATTACHMENT  IX. 

The  findings  of  the  EQRO's  documentation  review,  interviews  and  any  needed  demonstrations  of 
processes  should  be  documented  on  a  Denominator  Validation  Findings  -  Reviewer  Worksheet, 
such  as  that  located  at  ATTACHMENT  X. 


ONSITE  ACTIVITY  4:      Assess  processes  used  to  produce  numerators. 

The  focus  of  numerator  validation  is  on  determining  whether  the  MCO/PHP  has  correctly 
identified  and  evaluated  qualifying  medical  events  (e.g.,  diagnoses,  procedures,  and 
prescriptions)  in  order  to  include  appropriate  events  in  the  numerator  of  the  performance 
measure.  These  "medical  events"  may  be  identified  through  membership/enrollment  data, 
claim/encounter  data,  and/or  provider  data.  They  may  also  be  identified  through  data  extracted 
from  medical  records,  or  through  a  combination  of  both  administrative  data  and  medical  record 
abstraction,  i.e.,  the  "hybrid"  methodology. 

As  with  denominators,  accurate  and  complete  data  collection  is  vital  to  this  element  of 
performance  measure  calculation.  For  measures  that  include  sampling  in  the  methodology,  the 
entire  at-risk  population  must  have  an  equal  chance  to  be  included  in  the  numerator.  For  some 
measures,  particularly  those  frequently  focused  on  women  and  children  in  the  Medicaid 
population,  the  member  may  have  received  the  specified  service  outside  of  the  MCO/PHP 
provider  base  (e.g.,  children  receiving  immunizations  through  public  health  services  or  schools), 
so  an  effort  must  be  made  to  include  these  events  in  the  numerator. 
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If  either  medical  record  review  or  the  hybrid  methodology  is  used  to  calculate  the  performance 
measure,  the  EQRO  will  need  to  review  a  sample  of  medical  records  which  are  identified  as 
having  been  included  in  the  sample  drawn  by  the  MCO/PHP.  Following  specific  rules  and 
guidelines,  the  EQRO  will  determine  the  extent  to  which  data  obtained  from  medical  records  and 
noted  as  being  part  of  the  numerator  results  can  be  confirmed  during  medical  record  review 
validation  activities. 

During  this  activity,  for  each  performance  measure  calculated  by  the  MCO/PHP  and  chosen  to 
be  included  in  the  validation  activity,  the  EQRO  will  assess  the  extent  to  which: 

1 .  the  MCO/PHP  has  used  the  appropriate  data,  including  linked  data  from  separate  data 
sets,  to  identify  the  entire  at-risk  population  that  meets  the  specified  criteria  for  inclusion 
in  the  numerator. 

2.  the  MCO/PHP  has  adopted  and  followed  procedures  to  capture  data  for  those 
performance  measures  which  could  be  easily  under-reported  due  to  the  availability  of 
services  outside  the  MCO/PHP. 

3.  the  MCO's/PHP's  use  of  codes  to  identify  medical  events  (such  as  diagnoses,  procedures, 
prescriptions,  etc.)  are  complete,  accurate,  and  specific  in  correctly  describing  what  has 
transpired  and  when.  In  particular,  the  EQRO  will  assess  the  extent  to  which  these  codes 
were  correctly  evaluated  when  classifying  members  for  inclusion  or  exclusion  in  the 
numerator. 

4.  the  MCO/PHP  has  avoided  or  eliminated  all  double-counted  members  or  numerator 
events. 

5.  any  non-standard  codes  used  by  the  MCO/PHP  are  mapped  to  standard  codes  in  a  manner 
that  is  consistent,  complete,  and  reproducible.  The  EQRO  will  assess  this  through  a 
review  of  the  programming  logic  or  a  demonstration  of  the  program. 

6.  the  MCO/PHP  has  adhered  to  any  time  parameters  required  by  the  specifications  of  the 
performance  measure  (i.e.,  that  the  measured  event  occurred  during  the  time  period 
specified  or  defined  in  the  performance  measure). 

7.  medical  record  reviews  and  abstractions  have  been  carried  out  in  a  manner  that  facilitates 
the  collection  of  complete,  accurate,  and  valid  data  by  ensuring  that: 

•  record  review  staff  have  been  properly  trained  and  supervised  for  the  task. 

•  record  abstraction  tools  require  the  appropriate  notation  that  the  measured 
event  occurred. 

•  record  abstraction  tools  require  notation  of  the  results  or  findings  of  the 
measured  event  (if  applicable). 

8.  data  included  in  the  record  extract  files  are  consistent  with  data  found  in  the  medical 
records  for  a  sample  of  medical  records  for  applicable  performance  measures. 

9.  the  process  of  integrating  administrative  data  and  medical  record  data  for  the  purpose  of 
determining  the  numerator  is  consistent  and  valid. 
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Policies,  procedures,  data,  and  information  to  be  reviewed  in  conducting  these  activities  are  listed 
in  ATTACHMENT  XI.  These  activities  will  need  to  be  carried  out  with  respect  to  each 
performance  measure  calculated  by  the  MCO/PHP  and  included  in  the  EQRO  validation 
activities.  Because  of  this,  the  EQRO  will  need  to  refer  to  the  specifications  for  each  measure 
that  were  noted  by  the  EQRO  during  Pre-Onsite  activities  as  illustrated  in  ATTACHMENT  I.  In 
addition,  for  at  least  three  of  the  performance  measures  calculated  via  medical  record  review  or 
hybrid  methodology,  the  EQRO  will  need  to  validate  the  results  of  the  medical  record  review  for 
30  enrollees  who  were  found  to  meet  numerator  requirements  for  each  of  the  three  or  more 
measures.  Procedures  and  sample  tools  for  validating  medical  record  review  findings  are 
included  as  ATTACHMENT  XII. 

Information  obtained  from  a  review  of  policies,  procedures,  data,  and  information  should  be 
supplemented  or  confirmed,  as  needed,  though  interviews  with  MCO/PHP  personnel,  including: 
the  Director  of  Health/Medical  Information  Systems,  system  programmers  or  operators,  and 
selected  sub-contractors.  Suggested  questions  are  the  same  as  those  asked  with  respect  to 
denominators  and  are  located  at  ATTACHMENT  IX. 

The  findings  of  the  EQRO's  documentation  review,  interviews,  any  needed  demonstrations  of 
processes,  and  validation  of  medical  record  review  should  be  documented  on  a  Numerator 
Validation  Findings  -  Reviewer  Worksheet  such  as  that  located  at  ATTACHMENT  XIII. 


ONSITE  ACTIVITY  5:      Assess  the  sampling  process  (for  measures  NOT  calculated 

through  administrative  data). 

The  basic  task  in  validating  the  sampling  methodology  is  determining  whether  the  sample  validly 
reflects:  1)  the  performance  of  all  practitioners  and  providers  who  serve  Medicaid  enrollees  and 
whose  activities  are  the  subject  of  the  performance  measure;  and  2)  the  care  given  to  the  entire 
population  (including  special  populations  with  complex  care  needs)  to  which  the  performance 
measure  is  relevant. 

As  in  the  previous  activity  of  validating  the  population  included  in  a  denominator,  the  sampling 
methodology  employed  should  not  exclude  any  population  subgroups  to  which  the  topic  area  and 
performance  measure  apply.  For  example,  when  studying  well  child  care,  an  MCO's/PHP's 
sample  should  not  exclude  children  with  special  health  care  needs  whose  primary  care  provider  is 
a  specialist  other  than  a  pediatrician  or  family  practitioner. 

During  this  activity,  the  EQRO  will  assess  the  extent  to  which: 

1 .        the  sampling  methodology  used  by  the  MCO/PHP  produced  an  unbiased  sample  which  is 
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representative  of  the  entire  at-risk  population. 

2.  each  relevant  enrollee  or  provider  had  an  equal  chance  of  being  selected;  no  enrollees 
were  systematically  excluded  from  the  sampling. 

3.  the  MCO/PHP  followed  the  specifications  set  forth  by  the  State  for  the  performance 
measure  regarding  the  treatment  of  sample  exclusions  and  replacements  and,  if  any 
activity  took  place  involving  replacements  of  or  exclusions  from  the  sample,  the 
MCO/PHP  kept  adequate  documentation  of  that  activity. 

4.  each  provider  serving  a  given  number  of  enrollees  had  the  same  probability  of  being 
selected  as  any  other  provider  serving  the  same  number  of  enrollees. 

,5.        the  MCO/PHP  examined  its  sampled  files  for  bias  and  if  any  bias  was  detected,  the 
MCO/PHP  is  able  to  provide  documentation  that  describes  efforts  taken  to  correct  it. 

6.  the  sampling  methodology  treated  all  measures  independently  and  there  is  no  correlation 
between  drawn  samples.  (This  is  not  intended  to  be  a  validation  of  the  prescribed 
sampling  methodology  included  in  the  performance  measure  specifications,  because  the 
assumption  is  that  it  is  a  valid  methodology.  The  EQRO  validation  efforts  will  focus  on 
the  MCO's/PHP's  implementation  of  that  sampling  methodology  to  assess  the  extent  to 
which  it  has  correctly  followed  the  sampling  specifications.) 

7.  relevant  members  or  providers  who  were  not  included  in  the  sample  for  the  baseline 
measurement  have  the  same  chance  of  being  selected  for  the  follow-up  measurement  as 
those  who  were  included  in  the  baseline. 

8.  the  MCO/PHP  has  policies,  procedures,  and  documentation  that  files  from  which  the 
samples  were  drawn  are  maintained  so  that  if  the  sample  must  be  re-drawn,  or 
replacements  made,  the  original  population  is  intact. 

9.  the  sample  selected  conforms  to  the  methodology  set  forth  in  the  performance  measure 
specifications  and  the  sample  is  representative  of  the  entire  population. 

10.  sample  sizes  meet  the  requirements  of  the  performance  measure  specifications. 

11.  the  MCO/PHP  appropriately  handled  the  documentation  and  reporting  of  the  measure  if 
the  requested  sample  size  exceeds  the  population  size. 

12.  the  MCO/PHP  properly  oversampled  in  order  to  accommodate  potential  exclusions. 

13.  the  MCO/PHP  followed  proper  substitution  methodology  in  medical  record  review  (for 
measures  using  the  hybrid  methodology  or  medical  record  review). 

•  substitution  applied  only  to  those  members  who  met  the  exclusion  criteria 
detailed  in  the  performance  measure  specifications. 

•  substitutions  were  made  for  properly  excluded  records  and  the  percentage 
of  substituted  records  was  documented. 

Policies,  procedures,  data,  and  information  to  be  reviewed  in  conducting  these  activities  are  listed 
in  ATTACHMENT  XIV.  These  activities  need  to  be  carried  out  with  respect  to  each 
performance  measure  that  was  calculated  using  a  sample.  Because  of  this,  the  EQRO  will  need  to 
refer  to  the  "List  of  Performance  Measures  Calculated  by  the  MCO/PHP"  (TABLE  3)  and  the 


Validating  Performance  Measures  -  HCFA/CMSO/QPMG/DQSM  -  DRAFT-  8/11/00 


28 


specifications  for  each  measure  that  were  noted  by  the  EQRO  during  Pre-Onsite  activities  as 
illustrated  in  ATTACHMENT  I. 


Information  on  sampling  obtained  from  a  review  of  policies,  procedures,  data,  and  information 
should  be  supplemented  and  confirmed,  as  needed,  though  interviews  with  MCO/PHP  personnel, 
such  as:  the  Director  of  Health/Medical  Information  Systems,  system  programmers  or  operators, 
and  selected  sub-contractors.  Suggested  questions  to  ask  are  those  previously  identified  and 
included  as  ATTACHMENT  IX.  Validation  findings  regarding  sampling  should  be  documented 
on  a  worksheet  such  as  that  found  as  ATTACHMENT  XV. 


ONSITE  ACTIVITY  6:      Assess  submission  of  required  performance  measure  reports  to 

the  State. 

Once  the  MCO/PHP  calculates  the  required  performance  measures,  it  must  report  them  to  the 
State  in  the  manner  prescribed  by  the  State.  This  includes  reporting  the  measures  in  a  proper 
format,  whether  through  the  use  of  a  hardcopy  "shell"  report,  in  an  electronic  medium  and 
format,  or  some  combination  of  both.  During  the  Pre-Onsite  phase  of  the  review,  the  EQRO 
familiarized  itself  with  the  State's  format  and  reporting  requirements  for  the  MCO's/PHP's 
performance  measures.  During  this  activity,  the  EQRO  will  assess  whether  measures  were 
reported  to  the  State  in  the  manner  and  form  prescribed  by  the  State.  These  activities  will  need  to 
be  carried  out  with  respect  to  each  performance  measure  to  be  calculated  by  the  MCO/PHP. 
Because  of  this,  the  EQRO  will  need  to  refer  to  the  reporting  specifications  for  all  of  the 
measures  that  were  noted  by  the  EQRO  during  Pre-Onsite  activities  as  documented  in  TABLE  1 . 

To  assess  the  submission  of  required  performance  measure  reports  to  the  State,  the  EQRO  will 
review: 

•  procedures  for  submitting  reports  that  meet  State  requirements  (e.g.,  specified  electronic 
format,  supporting  documentation,  timing);  and 

•  documentation  that  procedures  for  properly  submitting  required  reports  to  State  were 
implemented  appropriately. 

The  extent  to  which  the  MCO/PHP  reported  the  calculated  performance  measures  to  the  State  in 
the  manner  and  form  prescribed  by  the  State  should  be  documented  in  the  EQRO's  report  to  the 
State. 
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POST-ONSITE  A  CTMTIES 


Objectives  for  Post-Onsite  Activities: 

The  EQRO  will  evaluate  all  gathered  information  and  submit  a  report  on  its  validation  findings 
to  the  State  following  either  Option  1  or  Option  2  below. 

Option  1 :      The  EQRO  submits  its  report  of  validation  findings  to  the  State  after  review  by 
the  MCO/PHP  for  any  factual  errors  or  omissions. 

OPTION  2:       The  EQRO  submits  a  final  report  to  the  State  after  providing  the  MCO/PHP  with 
the  opportunity  to  make  corrections  to  performance  measures  in  response  to 
preliminary  EQRO  findings.  This  would  occur  as  follows: 

•  The  EQRO  submits  to  the  MCO/PHP  a  preliminary  report  detailing  areas  of 
concern  and  suggested  methods  for  correction. 

•  After  allowing  the  MCO/PHP  to  correct  (as  practical)  any  problems  in  calculating 
or  reporting  performance  measures  that  were  identified  in  the  preliminary  report, 
the  EQRO  re-validates  selected  performance  measures  and  the  measurement 
processes. 

•  The  EQRO  again  evaluates  gathered  information  and  prepares  a  final  report  for 
the  State. 

•  The  EQRO  submits  its  report  of  validation  findings  to  the  State. 


POST-ONSITE  ACTIVITY  1:       Determine  preliminary  validation  findings  for  each 

measure. 

Once  the  EQRO  concludes  its  onsite  activities,  it  aggregates  the  validation  activity  findings  for 
each  performance  measure.  This  involves  review  and  analysis  of  findings  and  worksheets 
produced  for  each  performance  measure  selected  for  validation  and  for  the  MCO's/PHP's  IS  as  a 
result  of  Pre-Onsite  and  Onsite  activities.  In  particular,  these  include: 

•         Completed  performance  measure  validation  worksheets  for  each  performance  measure  to 
be  validated  (as  in  ATTACHMENT  I)  in  conjunction  with  the  Denominator  Validation 
Findings  (ATTACHMENT  X)  and  Numerator  Validation  Findings  (ATTACHMENT 
XIII). 
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•  For  measures  calculated  through  medical  record  review,  including  the  hybrid 
methodology,  the  completed  Medical  Record  Review  Validation  Tool  (ATTACHMENT 
XII). 

•  Findings  regarding  the  MCO's/PHP's  data  integration  and  control  procedures 
(ATTACHMENT  V);  and 

Sampling  validation  findings  (ATTACHMENT  XV). 

The  report  of  preliminary  validation  findings  identifies  any  areas  of  concern  for  each  of  the 
performance  measures  that  were  validated  by  the  EQRO  and  makes  suggestions  for 
improvement.  In  particular,  the  report  indicates  precisely  which  elements  of  the  plan-level 
performance  measures  were  invalid  (if  any).  This  information  provides  the  MCO/PHP  with 
specific  targets  for  correction  and  a  tool  that  can  be  used  to  focus  MCO/PHP  personnel  on  the 
changes  necessary  to  improve  the  production  process.  In  addition  to  communicating  in  writing, 
the  EQRO  may  participate  in  meetings  with  key  MCO/PHP  personnel  responsible  for  the 
calculation  and  reporting  of  performance  measures. 

Once  the  EQRO  has  submitted  its  preliminary  findings  to  the  MCO/PHP,  there  are  two  courses 
of  action  that  the  State  may  have  its  EQRO  pursue  with  respect  to  allowing  the  MCO/PHP  to 
respond  to  the  EQRO's  preliminary  findings: 

OPTION  1 :     The  MCO/PHP  may  offer  comments  and  documentation  to  support  correction  of 
factual  errors  and  omissions  in  the  EQRO's  preliminary  report;  or 

OPTION  2:     The  MCO/PHP  would  be  allowed  to  recalculate  performance  measures  based  on 
the  findings  of  the  EQRO.  The  EQRO  would  then  revalidate  the  revised 
performance  measure(s). 

Allowing  MCOs/PHPs  to  recalculate  measures  provides  States  and  Medicaid 
beneficiaries  with  a  greater  amount  of  accurate  information  on  MCO/PHP 
performance.  However,  this  option  requires  greater  time  and  financial  resources 
on  the  part  of  the  States,  EQROs  and  MCOs/PHPs.  If  Option  2  is  chosen  by  the 
State,  depending  on  the  extent  of  the  corrections  necessary  or  assistance  that  the 
MCO/PHP  needs  to  improve  its  performance  measure  production  processes,  the 
EQRO  schedules  a  time  to  re- visit  the  MCO/PHP  as  soon  as  practical,  in  order  to 
re-evaluate  the  performance  measures  before  they  are  reported  to  the  State.  This 
re-evaluation  follows  the  same  format  and  activities  as  the  initial  onsite  visit, 
except  that  the  EQRO  may  focus  only  on  those  activities  that  were  found  to  be 
problematic  during  the  first  validation  effort.  The  EQRO  will  use  worksheets  and 
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tools  that  are  identical  to  those  used  in  the  first  onsite  visit;  any  areas  not  re- 
reviewed  should  be  noted  accordingly. 

Once  Option  1  or  Option  2  is  completed,  and  the  MCO's/PHP's  comments  or  revised 
performance  measures  validation  findings  have  been  appropriately  incorporated  into  the 
validation  findings,  the  EQRO  will  submit  its  findings  to  the  State. 


POST-ONSITE  ACTIVITY  2:       Submission  of  validation  report  to  State. 

A  State  may  choose  one  of  two  options  for  determining  the  validity  of  each  of  the  MCO's/PHP's 
performance  measures: 

OPTION  1 :       The  EQRO  submits  all  working  papers  and  a  summary  of  findings  to  the  State. 

The  State  would  make  the  final  decision  on  the  validity  of  each  performance 
measure  and  compliance  with  reporting  requirements. 

Option  2:      The  EQRO  references  a  clearly  defined  set  of  decision  rules  for  determining  if 

each  of  the  MCO's/PHP's  reported  performance  measures  were  sufficiently  valid; 
i.e.,  accurate  and  complete.  In  this  instance,  the  State  would  still  receive  the  final 
report  and  all  supporting  documentation  and  would  have  the  final  authority  to 
determine  acceptable  validity  and  compliance  with  State  conditions. 

Regardless  of  which  option  a  State  chooses,  the  decision  rules  for  compliance  should  be  uniform 
across  MCOs/PHPs  within  the  State.  Because  States  may  differ  substantially  regarding  their 
requirements  for  Medicaid  MCOs/PHPs,  this  protocol  provides  a  framework  which  the  State  can 
use  with  its  own  specific  "percentage  rules"  or  requirements  for  determining  validity  of 
performance  measures. 

The  State  will  need  to  specify  the  level  of  bias  that  is  permissible  or  allowable  in  the  calculated 
performance  measures  in  order  for  an  MCO's/PHP's  performance  measure  to  be  considered 
"valid  measures."  Levels  currently  in  use  within  the  industry  range  from  5  percent  to  10  percent 
for  commercial  and/or  Medicare  product  lines.  Bias  in  reported  rates  can  result  from  many 
factors;  e.g.,  sampling  bias,  coding  errors,  and  in  particular,  problems  with  in  complete  data.  For 
example,  is  a  measure  calculated  using  a  data  set  that  is  known  to  be  only  50  percent  complete  a 
valid  measure  of  performance?  What  about  a  measure  using  data  that  is  75  or  85  percent 
complete?  Because  there  is  currently  no  generally  accepted  standard  for  data  completeness  in  the 
industry,  each  State  must  specify  the  extent  of  data  incompleteness  it  allows  in  measures  before 
the  measure  is  considered  to  be  "not  valid."  Data  completeness  was  addressed  as  part  of  the 
Performance  Measure  Validation  Worksheet  for  each  performance  measure  as  illustrated  in 
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ATTACHMENT  I.  The  EQRO  will  need  to  make  an  estimate  about  the  cumulative  affect  of  all 
sources  of  bias  on  the  validity  of  the  performance  measure. 

The  format  for  the  final  report  should  follow  the  format  specified  by  the  State,  but  should  include 
the  following  elements: 

•  a  list  of  measures  for  validation.  (It  is  possible  that  an  MCO/PHP  would  be  unable  to 
report  on  all  required  measures  for  reasons  that  would  be  explained  to  the  EQRO  and  the 
State.) 

•  a  description  of  the  onsite  validation  activities  including:  1)  a  list  of  the  EQRO's  team 
members  2)  a  description  of  the  pre-audit  strategy  and  considerations,  3)  a  description  of 
the  technical  methods  of  data  collection  and  analysis  used  by  the  EQRO,  4)  a  list  of 
interviewees,  and  5)  any  other  facts  relevant  to  the  onsite  process. 

•  details  and  results  of  the  validation  process  for  each  performance  measure,  including  any 
medical  record  abstractions  conducted. 

•  as  directed  by  the  State,  the  validation  findings  for  each  performance  measure  included  in 
the  EQRO  validation  activities. 

•  as  directed  by  the  State,  analysis  and  findings  with  respect  to  the  MCO's/PHP's  data 
integration  and  control  procedures  and  performance  measure  calculation  documentation. 

In  addition  to  reporting  to  the  State  on  the  extent  to  which  the  MCO/PHP  correctly  implemented 
processes  to  calculate  and  report  individual  plan-level  performance  measures,  other  aspects  of 
MCO/PHP  performance  measurement  that  the  State  may  want  the  EQRO  to  address  in  its  final 
report  include  the  extent  to  which  the  MCO/PHP  has: 

•  adequate  data  integration  and  control  necessary  for  accurate  reporting  of  performance 
measures;  and 

•  complete  and  accurate  documentation  of  data  and  processes  used  to  calculate  and  report 
performance  measures. 

In  addition,  the  EQRO  might  also  be  asked  to  submit  all  of  its  worksheets  and  tools  as  supporting 
documentation  to  the  report. 

END  OF  PROTOCOL 
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ATTACHMENT  I 


Example  of  a  Completed  Performance  Measure  Validation  Worksheet4 

Below  is  an  example  of  a  completed,  customized  performance  measure  validation  worksheet  similar  to  what  an 
EQRO  would  prepare  prior  to  its  onsite  visit.  This  worksheet  assumes  that  the  State  has  adopted  the  HEDIS 
methodology  for  this  performance  measure.  One  of  the  following  scoring  designations  must  be  checked  for  each 
audit  element: 

MET:  The  MCO's/PHP's  measurement  and  reporting  process  was  fully  compliant  with  State 
specifications. 

NOT  MET:  The  MCO's/PHP's  measurement  and  reporting  process  was  not  compliant  with  State 
specifications.  This  designation  should  be  used  for  any  audit  element  that  deviates  from  the  State 
specifications,  regardless  of  the  impact  of  the  deviation  on  the  final  rate.  All  audit  elements  with 
this  designation  must  include  explanation  of  the  deviation  in  the  comments  section. 

N/A:  The  audit  element  was  not  applicable  to  the  MCO's/PHP's  measurement  and  reporting 
process. 


PERFORMANCE  MEASURE  TO  BE  VALIDATED:  BREAST  CANCER  SCREENING 


METHODOLOGY  FOR  CALCULATING 
MEASURE:  (Check  one) 

ADMINISTRATIVE 

MEDICAL 

RECORD 

REVIEW 

HYBRH) 

AUDIT 
ELEMENTS 

AUDIT  SPECIFICATIONS 

MET 

NOT 
MET 

N/A 

COMMENTS 

DENOMINATOR 

1 .  Population 

•  Medicaid  population  appropriately 
segregated  from  commercial  / 
Medicare. 

•  Population  defined  as  effective 
Medicaid  enrollment  as  of  Dec.  3 1 , 
2000. 

•  Dual  Medicaid  and  Medicare 
beneficiaries  are  included. 

2.  Geographic  Area 

•     Includes  only  those  Medicaid 
enrollees  served  in  the 
MCO's/PHP's  reporting  area. 

'This  worksheet  is  adapted  from  the  IPRO  tools  used  in  the  audit  of  the  1997  Medicare  HEDIS  data. 
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AUDIT 
ELEMENTS 

AUDIT  SPECIFICATIONS 

MET 

NOT 
MET 

N/A 

COMMENTS 

3.  Age  &  Sex 

*  Members  aged  52-69  as  of 
12/31/00  (i.e.,  bora  between  1/1/31 
&  12/31/48) 

*  Only  females  selected 

4. Enrollment 
Calculation 

•  Were  members  of  plan  on 
12/31/00 

•  Were  continuously  enrolled  from 
1/1/99  to  12/31/00  with  one  break 
per  year  of  up  to  45  days  allowed. 

•  Switches  between  populations 
(Medicare,  Medicaid,  and 
commercial)  were  not  counted  as 
breaks. 

5. Data  Quality 

*     Based  on  the  IS  assessment 
findings,  are  any  of  the  data 
sources  for  this  denominator 
inaccurate? 

6. Proper  Exclusion 
Methodology  in 
Administrative  Data 
(If  no  exclusions 
were  taken,  check 

XT  /  A  \ 

N/A) 

•  Only  members  with 
contraindications  or  data  errors 
were  excluded. 

•  Contraindication  exclusions  were 
performed  according  to  current 
State  specifications. 

•  Only  the  codes  listed  in 
specifications  as  defined  by  State 
were  counted  as  contraindications. 

NUMERATOR 

7.  Administrative 
Data:  Counting 
Clinical  Events 

•  Standard  codes  listed  in  State 
specifications  or  properly  mapped 
internally  developed  codes  were 
used.  (Intended  to  reference 
appropriate  specifications  as 
defined  by  State.) 

*  Members  were  counted  only  once; 
double  counting  of  mammograms 
was  prevented. 

8.  Medical  Record 
Review 

Documentation 
Standards 

*  Record  abstraction  tool  required 
notation  of  the  date  that  the 
mammogram  was  performed. 

*  Record  abstraction  tool  required 
notation  of  the  mammogram  result 
or  finding. 

9.  Time  Period 

•     Mammogram  performed  on  or 
between  1/1/99  &  12/31/00. 

10. Data  Quality 

•     Properly  identified  enrollees. 
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*     Based  on  the  IS  assessment 
findings,  were  any  of  the  data 
sources  used  for  this  numerator 

lndLLUXalc : 

SAMPLING 

IF  ADMINISTRATIVE  METHOD  WAS  USED,  CHECK  "N/A"  FOR  AUDIT 
ELEMENTS  12, 13,  AND  14.. 

AUDIT 
ELEMENTS 

AUDIT  SPECIFICATIONS 

MET 

NOT 
MET 

N/A 

COMMENTS 

12.  Unbiased 
Sample 

•     As  specified  in  State 

specifications,  systematic  sampling 
method  was  utilized. 

13.  Sample  Size 

•     After  exclusions,  sample  size  is 
equal  to  1)  41 1,  2)  the 
appropriately  reduced  sample  size, 
which  used  the  current  year's 
administrative  rate  or  preceding 
year's  reported  rate,  or  3)  the  total 
population. 

14.  Proper 
Substitution 
Methodology  in 
Medical  Record 
Review  (If  no 
exclusions  were 
taken,  check  NA) 

*  Only  excluded  members  for  whom 
medical  record  review  revealed  1 ) 
contraindications  that  correspond 
to  the  codes  listed  in  appropriate 
specifications  as  defined  by  State 
or  2)  data  errors. 

*  Substitutions  were  made  for 
properly  excluded  records  and  the 
percentage  of  substituted  records 
was  documented. 

ADDITIONAL  QUESTIONS 


QUESTIONS 

YES 

NO 

Were  members  excluded  for  contraindications  found  in  the  administrative 
data? 

Were  members  excluded  for  contraindications  found  during  the  medical  record 
review? 

Were  internally  developed  codes  used? 

What  range  defines  the  impact  of  data  incompleteness  for  this  measure?  (Check  one.) 

0-5  percentage  points 

>5  -  1 0  percentage  points 

>10  -  20  percentage  points 
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>20  -  40  percentage  points 

>40  percentage  points 

Unable  to  Determine 

What  is  the  direction  of  the  bias?  Check  one:  OVER-REPORTING 

UNDER-REPORTING 

Upon  what  documentation  is  the  above  percentage  based?  (e.g.,  internal  reports,  studies, 
comparison  to  medical  records,  etc. 

VALIDATION  FINDING 

The  validation  finding  for  each  measure  is  determined  by  the  magnitude  of  the  errors  detected  for 
the  audit  elements,  not  by  the  number  of  audit  elements  determined  to  be  "NOT  MET." 
Consequently,  it  is  possible  that  an  error  for  a  single  audit  element  may  result  in  a  designation  of 
"NV"  because  the  impact  of  the  error  biased  the  reported  performance  measure  by  more  than  "x  " 
percentage  points.  Conversely,  it  is  also  possible  that  several  audit  element  errors  may  have  little 
impact  on  the  reported  rate  and,  thus  the  measure  could  be  given  a  designation  of  "SC."  The 
following  is  a  list  of  the  validation  findings  and  their  corresponding  definitions: 

FC      =        Fully  Compliant 

Measure  was  fully  compliant  with  State  specifications. 

SC      =        Substantially  Compliant 

Measure  was  substantially  compliant  with  State  specifications  and  had  only  minor 
deviations  that  did  not  significantly  bias  the  reported  rate. 

NV     =        Not  Valid 

Measure  deviated  from  State  specifications  such  that  the  reported  rate  was 
significantly  biased.  This  designation  is  also  assigned  to  measures  for  which  no 
rate  was  reported,  although  reporting  of  the  rate  was  required. 

NA      =        Not  Applicable 

Measure  was  not  reported  because  MCO/PHP  did  not  have  any  Medicaid 
enrollees  that  qualified  for  the  denominator. 


AUDIT  DESIGNATION 
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ATTACHMENT  II 


Potential  Documents  and  Processes  for  Review 

In  order  to  assess  the  MCO's/PHP's  information  system  and  the  validity  of  reported  performance 
measures,  the  EQRO  will  need  to  review  a  number  of  data  sources  and  processes.  The 
MCO/PHP  should  ensure  that  the  following  documents,  data,  and  procedures  are  available  to  the 
EQRO  for  observation;  the  EQRO  will  use  its  discretion  in  selecting  which  ones  to  review. 

Integration  and  Control  of  Data 

□  Procedures  and  standards  for  all  aspects  of  the  data  repository  (ies)  used  in  the  production 
of  performance  measures,  including  building,  maintaining,  managing,  testing,  and 
production  of  performance  measures. 

□  Manuals  covering  application  system  development  methodology,  database  development, 
and  design  and  decision  support  system  utilization. 

□  Control  system  documentation  including  flow  charts  and  codes  for  backups,  recovery, 
archiving,  and  other  control  functions. 

□  Procedures  to  consolidate  information  from  disparate  transaction  files. 

□  Record  and  file  formats  and  descriptions,  for  entry,  intermediate,  and  repository  files. 

□  Electronic  formats  and  protocols. 

□  Electronic  transmission  procedures  documentation. 

□  Processes  to  extract  information  from  the  repository(ies). 

□  Source  code  data  entry,  data  transfer,  and  data  manipulation  programs  and  processes. 

□  Descriptive  documentation  for  data  entry,  transfer,  and  manipulation  programs  and 
processes. 

□  If  applicable,  procedures  for  coordinating  activities  of  multiple  subcontractors  in  a  way  that 
safeguards  the  integrity  of  the  performance  measurement  data. 

□  Samples  of  data  from  repository  and  transaction  files  to  assess  accuracy  and  completeness 
of  the  transfer  process. 

□  Comparison  of  actual  results  from  file  consolidation  and  data  abstracts  to  those  which 
should  have  resulted  according  to  documented  algorithms. 

□  Documentation  of  data  flow  among  vendors  to  assess  the  extent  to  which  there  has  been 
proper  implementation  of  procedures  for  coordinating  activities  to  safeguard  the  integrity  of 
the  performance  measure  data. 

□  Documentation  of  data  cutoff  dates. 

□  Documentation  of  proper  run  controls  and  of  staff  review  of  report  runs. 


Attachment  II 


38 


□  Copies  of  files  and  databases  used  for  performance  measure  calculation  and  reporting. 

□  Procedures  governing  production  process  for  plan-level  performance  measures,  including 
standards  and  schedules. 

Collection,  Calculation,  and  Documentation  of  Performance  Measurements 

□  Policies  which  stipulate  and  enforce  documentation  of  data  requirements,  issues,  validation 
efforts,  and  results. 

□  A  project  or  measurement  plan  for  each  performance  measure. 

□  Documentation  of  programming  specifications,  including  work  flow,  data  sources,  and  uses 
which  include  diagrammatic  or  narrative  descriptions. 

□  Documentation  of  the  original  universe  of  data  that  includes  record-level  patient  identifiers 
that  can  be  used  to  validate  entire  programming  logic  for  creating  denominators, 
numerators,  and  samples. 

□  Documentation  of  computer  queries,  programming  logic,  or  source  code  used  to  create  final 
denominators,  numerators,  and  interim  data  files. 

□  Documentation  that  includes  dated  job  log  or  computer  run  for  denominators  and 
numerators,  with  record  counts  for  each  programming  step  and  iteration. 

□  Documentation  of  medical  record  review  including:  qualifications  of  medical  record  review 
supervisor  and  staff;  reviewer  training  materials;  audit  tools  used,  including  completed 
copies  of  each  record-level  reviewer  determination;  all  case-level  critical  performance 
measure  data  elements  used  to  determine  a  positive  or  negative  event  or  exclude  a  case 
from  same;  and  inter-rater  reliability  testing  procedures  and  results. 

□  Documentation  of  results  of  statistical  tests  and  any  corrections  or  adjustments  to  data 
along  with  justification  for  such  changes. 

□  Documentation  of  sources  of  any  supporting  external  data  or  prior  years'  data  used  in 
reporting. 

□  Policies  to  assign  unique  membership  ID  that  allows  all  services  to  be  properly  related  to 
the  specific  appropriate  enrollee,  despite  changes  in  status,  periods  of  enrollment  or 
disenrollment,  or  changes  across  product  lines  (e.g.,  Medicare  and  Medicaid). 

□  Procedures  to  identify,  track,  and  link  member  enrollment  by  product  line,  product, 
geographic  area,  age,  sex,  member  months,  and  member  years. 

□  Procedures  to  track  individual  members  through  enrollment,  disenrollment,  and  possible  re- 
enrollment. 

□  Procedures  to  track  members  through  changes  in  family  status,  changes  in  benefits  or 
managed  care  type  (if  they  switch  between  Medicaid  coverage  and  another  product  within 
the  same  MCO/PHP). 
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□  Methods  to  define  start  and  cessation  of  coverage. 

□  Procedures  to  link  member  months  to  member  age. 

□  Description  of  software  or  programming  languages  used  to  query  each  database. 

□  Description  of  software  used  to  execute  sampling  sort  of  population  files  when  sampling 
(systematic)  is  used. 

□  Member  database. 

□  Provider  data  (including  facilities,  labs,  pharmacies,  physicians,  etc.). 

□  Database  record  layout  and  data  dictionary. 

□  Survey  data. 

□  Policies  to  maintain  files  from  which  the  samples  are  drawn  in  order  to  keep  population 
intact  in  the  event  that  a  sample  must  be  re-drawn,  or  replacements  made. 

□  Computer  source  code  or  logic  identifying  specified  sampling  techniques,  and 
documentation  that  the  logic  matches  the  specifications  set  forth  for  each  performance 
measure,  including  sample  size  and  exclusion  methodology. 

□  Methods  used  for  sampling  for  measures  calling  for  hybrid  data  (combination  of  medical 
records  and  administrative  data)  or  solely  medical  record  review. 

□  Documentation  assuring  that  sampling  methodology  treats  all  measures  independently  and 
that  there  is  no  correlation  between  drawn  samples. 

□  Observation  or  documentation  of  procedures  in  which  a  biased  sample  was  identified  and 
corrected. 

□  Documentation  of  "frozen"  or  archived  files  from  which  the  samples  were  drawn,  and  if 
applicable,  documentation  of  the  MCO's/PHP's  process  to  re-draw  a  sample  or  obtain 
necessary  replacements. 

□  For  performance  measures  which  are  easily  under-reported,  procedures  to  capture  data  that 
may  reside  outside  the  MCO's/PHP's  data  sets. 

□  Procedures  for  mapping  non-standard  codes  to  standard  coding  to  ensure  consistency 
completeness,  and  reproducibility. 

□  Policies,  procedures,  and  materials  that  evidence  proper  training,  supervision,  and  adequate 
tools  for  medical  record  abstraction  tasks.  (May  include  medical  record  abstraction  tools, 
training  material,  checks  of  inter-rater  reliability,  etc.) 

□  Procedures  for  assuring  that  combinations  of  record-review  data  with  administratively 
determined  data  are  consistent  and  verifiable. 

□  Evidence  that  MCO's/PHP's  use  of  codes  to  identify  medical  events  were  correctly 
evaluated  when  classifying  members  for  inclusion  or  exclusion  in  the  numerator. 

□  Evidence  that  MCO/PHP  has  counted  each  member  and/or  event  only  once. 
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□  Programming  logic  or  demonstration  that  confirms  that  any  non-standard  codes  used  in 
determining  the  numerator  have  been  mapped  to  a  standard  coding  scheme  in  a  manner  that 
is  consistent,  complete,  and  reproducible. 

□  Programming  logic  or  source  code  that  identifies  the  process  for  integrating  administrative 
and  medical  record  data  for  numerator. 

□  Procedures  for  properly  executing  complex  medical  algorithms,  such  as  claim-dependent 
events;  events  that  require  matching  claims  and  pharmacy  data;  events  that  require 
matching  visit  codes;  and  events  that  require  accurately  identifying  and  computing  multiple 
numerator  events. 

□  Procedures  for  displaying  denominator  counts,  numerator  counts,  precision  levels,  sums 
and  cross-totals. 

□  Procedures  for  reporting  small  sample  sizes  (to  be  consistent  with  required  methodology 
established  by  State). 

□  Programming  logic  and/or  source  code  for  arithmetic  calculation  of  each  measure. 

□  Review  of  reported  measures  to  assess  consistency  of  common  elements  (e.g.,  membership 
counts,  number  of  pregnancies  and  births,  etc.). 

□  Programming  logic  and/or  source  code  for  measures  with  complex  algorithms,  to  ensure 
adequate  matching  and  linkage  among  different  types  of  data. 

□  Documentation  showing  confidence  intervals  of  calculations  when  sampling  methodology 
used. 

□  Documentation  showing  calculation  of  levels  of  significance  of  changes. 

□  Procedures  for  submitting  reports  that  meet  State  requirements  (e.g.,  specified  electronic 
format,  supporting  documentation,  timing). 

□  Documentation  that  procedures  for  properly  submitting  required  reports  to  State  were 
implemented  appropriately. 
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ATTACHMENT  III     ( ^ 


IS  Data  Integration  and  Control  -  Documentation  Review 

Worksheet 


Documentation 

Reviewed 

Not 
Reviewed 

Comments 

Procedures  and  standards  for  all  aspects 
of  the  data  repository  (ies),  including 
building,  maintaining,  managing,  testing, 
and  production  of  performance  measures. 

Manuals  covering  application  system 
development  methodology,  database 
development  and  design,  and  decision 
support  system  utilization. 

Control  system  documentation  including 
flow  charts  and  codes  for  backups, 
recovery,  archiving,  and  other  control 
functions 

Procedures  to  consolidate  information 
from  disnarate  transaction  files  to  sunnort 
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performance  measurement. 

Record  and  file  formats  and  descriptions, 
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files. 

Electronic  formats  and  protocols. 

Electronic  transmission  procedures  ' 
documentation. 

Processes  to  extract  information  from  the 
repository  to  produce  intended  result. 

Source  code  data  entry,  data  transfer,  and 
data  manipulation  programs  and 
processes. 

Descriptive  documentation  for  data  entry, 
data  transfer,  data  manipulation  programs 
and  processes. 

If  applicable,  procedures  for  coordinating 
activities  of  multiple  subcontractors  in  a 
way  that  safeguards  the  integrity  of  the 
performance  measure  data. 
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Samples  of  data  from  repository  and 
transaction  files  to  assess  accuracy  and 
completeness  of  the  transfer  process. 

Comparison  of  actual  results  from  file 
consolidation  and  data  abstracts  to  those 
which  should  have  resulted  according  to 
documented  algorithms. 

Documentation  of  data  flow  among 
vendors  to  assess  the  extent  to  which 
there  has  been  proper  implementation  of 
procedures  for  coordinating  activities  to 
safeguard  the  integrity  of  the  performance 
measure  data. 

Documentation  of  data  cutoff  dates. 

Documentation  of  proper  run  controls 

anH  r»f  ctniF rpvipw  nf  rpnnrt  nin^ 
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Copies  of  files  and  databases  used  for 
performance  measure  calculation  and 

Procedures  governing  production  process 
of  plan-level  performance  measures, 
including  standards  and  schedules. 

In  the  comments  section,  be  sure  to  address  the  following: 

Compare  samples  of  data  in  the  repository  to  transaction  files.  Are  any  members,  providers,  or 
services  lost  in  the  process? 

Is  the  required  level  of  coding  detail  maintained  (e.g.,  all  significant  digits,  primary  and 
secondary  diagnoses  remain)? 

If  the  plan  uses  a  performance  measure  repository,  review  the  repository  structure.  Does  it 
contain  all  the  key  information  necessary  for  performance  measure  reporting? 

How  does  the  MCO/PHP  test  the  process  used  to  create  the  performance  measure  reports? 

Does  the  MCO/PHP  use  any  algorithms  to  check  the  reasonableness  of  data  integrated  to  report 
the  plan-level  performance  measures 
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Examine  report  production  logs  and  run  controls.  Is  there  adequate  documentation  oft 
performance  measure  report  generation  process?  How  are  report  generation  programs 
documented?  Is  there  a  type  of  version  control  in  place? 


ATTACHMENT  III 


ATTACHMENT  IV 


Guide  for  Interviews  of  MCO/PHP  Personnel  Concerning 
Data  Integration  and  Control 

Background  Information: 

Name  of  MCO/PHP: 

Date: 

Location: 

Year  of  First  Medicaid  Enrollment: 

Year  of  First  Plan-Level  Performance  Report: 

Auditors: 

Names  and  Titles  of  Individuals  Interviewed: 


Has  the  MCO/PHP  previously  undergone  an  audit  of  its  State  performance  measure  reporting 
process?  If  so,  when  did  the  audit  take  place  and  who  conducted  it? 


Other  general  issues: 
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Interview  Questions: 

1 .  How  is  performance  measure  data  collection  accomplished: 

•  By  querying  the  applicable  information  system  on-line? 

•  By  using  extract  files  created  for  analytical  purposes?  If  so,  how  frequently  are 
the  files  updated?  How  do  they  account  for  claim/encounter  submission  and 
processing  lags?  How  is  the  file  creation  process  checked  for  accuracy? 

•  By  using  a  separate  relational  database  or  data  warehouse?  If  so,  is  this  the  same 
system  all  other  reporting  is  produced  from?  Are  reports  created  from  a  vendor 
software  product?  If  so,  how  frequently  are  the  files  updated?  How  are  reports 
checked  for  accuracy? 

2.  Review  the  procedure(s)  for  consolidating  claims/encounter,  member,  provider,  and  other 
data  necessary  for  performance  reporting  (whether  it  be  into  a  relational  database  or  file 
extracts  on  a  measure-by-measure  basis). 

How  many  different  sources  of  data  are  merged  together  to  create  reports? 

•  What  control  processes  are  in  place  to  ensure  that  this  merger  is  accurate  and 
complete? 

3.  How  does  the  plan  test  the  process  used  to  create  the  performance  measure  reports? 

4.  Does  the  plan  use  any  algorithms  to  check  the  reasonableness  of  data  integrated  to  report 
the  plan-level  performance  measures 

5.  Are  performance  measurement  reporting  programs  reviewed  by  supervisory  staff? 

6.  Is  there  an  internal  backup  for  performance  measure  programmers  -  do  others  know  the 
programming  language  and  the  structure  of  the  actual  programs?  Is  there  documentation? 

7.  How  does  the  plan  prevent  loss  of  claim  and  encounter  data  when  systems  fail? 

8.  What  administrative  data  backup  systems  are  in  place? 

9.  What  types  of  authorization  are  required  to  be  able  to  access  claims/encounter,  provider, 
membership,  and  performance  measure  repository  data? 

Describe  Documentation  Review  and  Demonstrations  Provided: 
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ATTACHMENT  V 
Data  Integration  and  Control  Findings  -  Documentation  Worksheet 


Data  Integration  and  Control  Element 

Met 

Not  Met 

N/A 

Comments 

Accuracy  of  data  transfers  to  assigned  performance  measure  repository. 

•   MCO/PHP  processes  accurately  and 
completely  transfer  data  from  the 
transaction  files  (e.g.,  membership, 
provider,  encounter/claims)  into  the 
repository  used  to  keep  the  data  until  the 
calculations  of  the  performance  measures 
have  been  completed  and  validated. 

•   Samples  of  data  from  repository  are 
complete  and  accurate. 

Accuracy  of  file  consolidations,  extracts,  and  derivations. 

•   MCO's/PHP's  processes  to  consolidate 
diversified  files,  and  to  extract  required 
information  from  the  performance 
measure  repository  are  appropriate. 

•   Actual  results  of  file  consolidations  or 
extracts  were  consistent  with  those  which 
should  have  resulted  according  to 
documented  algorithms  or  specifications. 

•   Procedures  for  coordinating  the  activities 
of  multiple  subcontractors  ensure  the 
accurate,  timely,  and  complete  integration 
of  data  into  the  performance  measure 
database. 

•   Computer  program  reports  or 

documentation  reflect  vendor  coordination 
activities,  and  no  data  necessary  to 
performance  measure  reporting  are  lost  or 
inappropriately  modified  during  transfer. 

If  the  MCO/PHP  uses  one,  the  structure  and  format  of  the  performance  measure  data 
repository  facilitates  any  required  programming  necessary  to  calculate  and  report  required 
performance  measures. 

•   The  repository's  design,  program  flow 
charts,  and  source  codes  enable 
analyses  and  reports. 

•   Proper  linkage  mechanisms  have  been 
employed  to  join  data  from  all 
necessary  sources  (e.g.,  identifying  a 
member  with  a  given 
disease/condition). 
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Assurance  of  effective  management  of  report  production  and  of  the  reporting  software. 

•   Examine  and  assess  the  adequacy  of 
the  documentation  governing  the 
production  process,  including 
MCO/PHP  production  activity  logs, 
and  MCO/PHP  staff  review  of  report 
runs. 

•  Prescribed  data  cutoff  dates  were 
followed. 

•   The  MCO/PHP  has  retained  copies  of 
files  or  databases  used  for  performance 
measure  reporting,  in  the  event  that 
results  need  to  be  reproduced. 

•   Review  documentation  standards  to 
determine  the  extent  to  which  the 
reporting  software  program  is  properly 
documented  with  respect  to  every 
aspect  of  the  performance 
measurement  reporting  repository, 
including  building,  maintaining, 
managing,  testing,  and  report 
production. 

•   Review  the  MCO's/PHP's  processes 
and  documentation  to  determine  the 
extent  to  which  they  comply  with  the 
MCO/PHP  standards  associated  with 
reporting  program  specifications,  code 
review,  and  testing. 
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ATTACHMENT  VI 

Data  and  Processes  Used  to  Calculate  and  Report  Performance  Measures  - 

Documentation  Review  Worksheet 


Documentation 

Reviewed 

Not 
Reviewed 

Comments 

Policies  which  stipulate  and  enforce 
documentation  of  data  requirements,  issues, 
validation  efforts  and  results. 

Procedures  for  displaying  denominator 
counts,  numerator  counts,  precision  levels, 
sums,  and  cross-totals. 

Procedures  for  reporting  small  sample  sizes 
(to  be  consistent  with  required 
methodology  established  by  State). 

Review  of  reported  measures  to  assess 
consistency  of  common  elements  (e.g., 
membership  counts,  number  of  pregnancies 
and  births,  etc.). 

For  each  measure: 

Programming  logic  and/or  source  code  for 
arithmetic  calculation. 

A  project  or  measurement  plan,  including 
work  flow. 

Documentation  of  programming 
specifications  and  data  sources. 

Documentation  of  the  original  universe  of 
data  including  record-level  patient 
identifiers  that  can  be  used  to  validate 
entire  programming  logic  for  creating 
denominators,  numerators,  and  samples. 

Documentation  of  computer  queries, 
programming  logic,  or  source  code  used  to 
create  denominators,  numerators,  and 
interim  data  files. 

Documentation  that  includes  dated  job  log 
or  computer  run  for  denominators  and 
numerators,  with  record  counts  for  each 
programming  step  and  iteration. 
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Documentation 

Reviewed 

Not 
Reviewed 

Comments 

Documentation  of  medical  record  review 
for  each  measure,  as  appropriate, 
including:  qualifications  of  medical  record 
review  supervisor  and  staff;  reviewer 
training  materials;  audit  tools  used, 
including  completed  copies  of  each  record- 
level  reviewer  determination;  all  case-level 
critical  performance  measure  data  elements 
used  to  determine  a  positive  or  negative 
event  or  exclude  a  case  from  same;  and 
inter-rater  reliability  testing  procedures  and 
results. 

Documentation  of  results  of  statistical  tests 
and  any  corrections  or  adjustments  to  data 
along  with  justification  for  such  changes 
for  each  measure,  as  appropriate. 

Documentation  showing  calculation  of 
levels  of  significance  of  changes  for  each 
measure. 

Documentation  (for  each  performance 
measure,  as  appropriate)  showing 
confidence  intervals  of  calculations  when 
sampling  methodology  used. 

■ 

Documentation  of  sources  of  any 
supporting  external  data  or  prior  years' 
data  used  in  reporting  (for  each 
performance  measure,  as  appropriate). 

Describe  Documentation  Reviewed  and  Demonstrations  Provided: 
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ATTACHMENT  VII 


Data  and  Processes  Used  to  Calculate  and  Report  Performance  Measures  - 

Documentation  Worksheet 


Audit  Element 

Met 

Not  Met 

N/A 

Comments 

Measurement  plans  and  policies  which  stipulate  and  enforce  documentation  of  data 
requirements,  issues,  validation  efforts  and  results.  These  include: 

•     Data  file  and  field  definitions  used 
for  each  measure. 

•     Maps  to  standard  coding  if  not 
used  in  original  data  collection. 

•     Statistical  testing  of  results  and 
any  corrections  or  adjustments 
made  after  processing. 

Documentation  of  programming  specifications  (which  may  be  either  a  schematic  diagram  or 
in  narrative  form)  for  each  measure  includes  at  least  the  following: 

•     All  data  sources,  including 
external  data  (whether  from  a 
vendor,  public  registry,  or  other 
outside  source),  and  any  prior 
years'  data  (if  applicable). 

•     Detailed  medical  record  review 
methods  and  practices,  including 
the  qualifications  of  medical 
record  review  supervisor  and  staff; 
reviewer  training  materials;  audit 
tools  used,  including  completed 
copies  of  each  record-level 
reviewer  determination;  all  case- 
level  critical  performance  measure 
data  elements  used  to  determine  a 
positive  or  negative  event  or 
exclude  a  case  from  same;  and 
inter-rater  reliability  testing 
procedures  and  results. 

•     Detailed  computer  queries, 

programming  logic,  or  source  code 
used  to  identify  the  population  or 
sample  for  the  denominator  and/or 
numerator 
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Audit  Element 

Met 

Not  Met 

N/A 

Comments 

•  If  sampling  used,  description  of 
sampling  techniques,  and 
documentation  that  assures  the 
reviewer  that  samples  used  for 
baseline  and  repeat  measurements 
of  the  performance  measures  were 
chosen  using  the  same  sampling 
frame  and  methodology. 

•  Documentation  of  calculation  for 
changes  in  performance  from 
previous  periods  (if  applicable), 
including  statistical  tests  of 
significance. 

•  Data  that  are  related  from  measure  to 
measure  are  consistent  (e.g., 
membership  counts,  provider  totals, 
number  of  pregnancies  and  births). 

•  Appropriate  statistical  functions  are 
used  to  determine  confidence 
intervals  when  sampling  is  used  in 
the  measure. 

•  When  determining  improvement  in 
performance  between  measurement 
periods,  appropriate  statistical 
methodology  is  applied  to  determine 
levels  of  significance  of  changes. 

ATTACHMENT  VII 


52 


ATTACHMENT  VIII 


Policies,  Procedures,  Data  and  Information  Used  to  Produce  Denominators: 

Review  Worksheet 


Policies,  Procedures,  Data,  Information 
to  be  reviewed 

Reviewed 

Not  Reviewed 

Comments 

Policies  to  assign  unique  membership  ID  that 
allows  all  services  to  be  properly  related  to 
the  specific  appropriate  enrollee,  despite 
changes  in  status,  periods  of  enrollment  or 
disenrollment,  or  changes  across  product 
lines  (e.g.,  Medicare  and  Medicaid). 

Procedures  to  identify,  track,  and  link 
member  enrollment  by  product  line,  product, 
geographic  area,  age,  gender,  member 
months,  member  years. 

Procedures  to  track  individual  members 
through  enrollment,  disenrollment,  and 
possible  re-enrollment. 

Procedures  to  track  members  through 
changes  in  family  status,  changes  in 
employment  or  benefits  or  managed  care  type 
(if  they  switch  between  Medicaid  coverage 
and  another  product  within  the  same 
MCO/PHP). 

Methods  to  define  start  and  cessation  of 

Procedures  to  link  member  months  to 
member  age. 

Description  of  software  or  programming 
languages  used  to  query  each  database. 

Programming  logic  and/or  source  code  for 
arithmetic  calculation  of  each  measure. 

Programming  logic  and/or  source  code  for 
measures  with  complex  algorithms,  to  ensure 
adequate  matching  and  linkage  among 
different  types  of  data. 

Member  database. 

Provider  data  (including  facilities,  labs, 
pharmacies,  physicians,  etc.). 

Database  record  layout  and  data  dictionary. 

Survey  data. 
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ATTACHMENT  IX 


QUESTIONS  FOR  ASSESSING  PROCESSES 
USED  TO  PRODUCE  DENOMINATORS  AND  NUMERATORS 

1 .  If  any  part  of  your  network/data/membership  was  excluded  from  a  performance  measure, 
how  and  why  did  you  decide  to  exclude  it? 

2.  Why  did  you  select  the  reporting  methodology  (e.g.,  administrative,  or  hybrid)  used  to 
create  each  of  the  measures  (where  there  was  an  option)? 

3.  Did  you  use  the  State  technical  specifications  as  the  specifications  for  the  programmers, 
or  did  your  MCO/PHP  write  its  own  instructions/translations  for  the  programmers? 

4.  Are  there  any  manual  processes  used  for  calculating  denominators  and/or  numerators? 
Are  manual  processes  used  for  sampling? 


5.        Are  any  measures  calculated  by  vendors?  If  yes,  are  they  checked  for  accuracy?  Please 
describe. 


6.        Do  you  have  any  concerns  about  the  integrity  of  the  information  used  to  create  any  of  the 
measures?  Please  describe. 


7.        Do  you  know  of  any  deviations  from  performance  measure  specifications  that  were 
necessary  because  of  data  available  or  because  of  your  plan's  information  systems 
capabilities? 


Other  issues. 


Names  of  Individuals  Interviewed: 
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ATTACHMENT  X 


Denominator  Validation  Findings  -  Reviewer  Worksheet 


Audit  Element 

Met 

Not 
Met 

N/A 

1  Comments 

For  each  of  the  performance  measures,  all  members  of  the  relevant  populations  identified  in 
the  performance  measure  specifications  are  included  in  the  population  from  which  the 
denominator  is  produced 

All  members  who  were  eligible  to 
receive  the  specified  services  were 
included  in  the  initial  population  from 
which  the  final  denominator  was 
produced.  This  "at  risk"  population 
included  both  members  who  received 
the  services,  as  well  as  those  who  did 
not.  This  same  standard  applies  to 
provider  groups  or  other  relevant 
populations  identified  in  the 
specifications  of  each  performance 
measure. 

Adequate  programming  logic  or  source 
members  of  the  specified  denominator  i 

•  code  exists  to  appropriately  identify  all  "relevant" 
copulation  for  each  of  the  performance  measures. 

For  each  measure,  programming  logic 
or  source  code  which  identifies,  tracks, 
and  links  member  enrollment  within 
and  across  product  lines  (e.g., 
Medicare  and  Medicaid),  by  age  and 
sex,  as  well  as  through  possible  periods 
of  enrollment  and  disenrollment,  has 
been  appropriately  applied  according 
to  the  specifications  of  each 
performance  measure. 

• 

Calculations  of  continuous  enrollment 
criteria  were  correctly  carried  out  and 
applied  to  each  measure  (if 
applicable). 

Proper  mathematical  operations  were 
used  to  determine  patient  age  or  range. 

The  MCO/PHP  can  identify  the 
variable(s)  that  define  the  member's 
sex  in  every  file  or  algorithm  needed  to 
calculate  the  performance  measure 
denominator,  and  the  MCO/PHP  can 
explain  what  classification  is  carried 
out  if  neither  of  the  required  codes  is 
present. 
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Audit  Element 

Met 

Not 
Met 

N/A 

Comments 

Correct  calculation  of  member  months  and  member  years. 

The  MCO/PHP  has  correctly 
calculated  member  months  and 
member  years,  if  applicable  to  the 
performance  measure. 

Completeness  and  accuracy  of  the  codes  used  to  identify  medical  events  has  been  identified 
and  the  codes  have  been  appropriately  applied. 

The  MCO/PHP  has  properly  evaluated 
the  completeness  and  accuracy  of  any 
codes  used  to  identify  medical  events, 
such  as  diagnoses,  procedures,  or 
prescriptions,  and  these  codes  have 
been  appropriately  identified  and 
applied  as  specified  in  each 
performance  measure. 

Specified  time  parameters  are  followed 

Any  time  parameters  required  by  the 
specifications  of  the  performance 
measure  are  followed  (e.g.,  cut  off 
dates  for  data  collection,  counting  30 
calendar  days  after  discharge  from  a 
hospital,  etc.). 

Exclusion  criteria  included  in  the  performance  measure  specifications  have  been  followed 

Performance  measure  specifications  or 
definitions  that  exclude  members  from 
a  denominator  were  followed.  For 
example,  if  a  measure  relates  to  receipt 
of  a  specific  service,  the  denominator 
may  need  to  be  adjusted  to  reflect 
instances  in  which  the  patient  refuses 
the  service  or  the  service  is 
contraindicated. 

Systems  to  estimate  populations  which  cannot  be  accurately  counted  exist  and  are  utilized 
when  appropriate. 

Systems  or  methods  used  by  the 
MCO/PHP  to  estimate  populations 
when  they  cannot  be  accurately  or 
completely  counted  (e.g.,  newborns) 
are  valid. 
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ATTACHMENT  XI 


Policies,  Procedures,  Data,  and  Information  Used  to  Produce  Numerators: 

Review  Worksheet 


Documentation 

Reviewed 

Not  Reviewed 

Comments 

For  performance  measures  which  are 
easily  under-reported,  procedures  to 
capture  data  that  may  reside  outside  the 
MCO/PHP's  data  sets. 

Procedures  for  mapping  non-standard 
codes  to  standard  coding  to  ensure 
consistency,  completeness,  and 
reproducibility. 

Policies,  procedures,  and  materials  that 
evidence  proper  training,  supervision, 
and  adequate  tools  for  medical  record 
abstraction  tasks.  (May  include  medical 
record  abstraction  tools,  training 
material,  checks  of  inter-rater  reliability, 
etc.) 

Procedures  for  assuring  that 
combinations  of  record-review  data  with 
administratively  determined  data  are 
consistent  and  verifiable. 

MCO's/PHP's  use  of  codes  to  identify 
medical  events  were  correctly  evaluated 
when  classifying  members  for  inclusion 
or  exclusion  in  the  numerator. 

Evidence  that  MCO/PHP  has  counted 
each  member  and/or  event  only  once. 

Programming  logic  or  demonstration  that 
confirms  that  any  non-standard  codes 
used  in  determining  the  numerator  have 
been  mapped  to  a  standard  coding 
scheme  in  a  manner  that  is  consistent, 
complete,  and  reproducible. 

Programming  logic  or  source  code  that 
identifies  process  for  integrating 
administrative  and  medical  record  data 
for  numerator. 

Programming  logic  and/or  source  code 
for  arithmetic  calculation  of  each 
measure. 

 ■  i 

Programming  logic  and/or  source  code 
for  measures  with  complex  algorithms,  to 
ensure  adequate  matching  and  linkage 
among  different  types  of  data. 
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Describe  documentation  review  and  any  demonstrations  provided: 
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ATTACHMENT  XII 


Medical  Record  Review  Validation  Tools 

The  purpose  of  medical  record  review  validation  is  to  verify  the  accuracy  of  the  medical  record 
review  conducted  by  each  MCO/PHP.  For  each  of  at  least  three  measures  for  which  the  hybrid 
method  or  solely  medical  record  review  was  used,  the  EQRO  will  validate  the  medical  records  of 
30  enrollees  found  to  meet  numerator  requirements.  Only  those  members  included  in  a  hybrid  or 
solely  medical  record  review  sample  will  be  selected  -  the  EQRO  will  not  be  conducting  medical 
record  audits  to  validate  administrative  data.  Therefore,  if  an  MCO/PHP  used  only  administrative 
data  for  a  particular  measure,  that  measure  will  not  be  part  of  the  medical  record  review 
validation  process. 

For  each  measure  in  which  the  hybrid  method  or  solely  medical  record  review  was  used,  the 
EQRO  will  request  a  list  of  all  of  the  members  in  the  MCO's/PHP's  medical  record  review 
sample.  From  that  list  the  EQRO  will  identify  a  sample  of  30  members  who  meet  numerator 
requirements.  MCOs/PHPs  will  then  be  asked  to  provide  access  to  or  copies  of  medical  records 
so  that  the  EQRO  can  verify  that  each  member  was  appropriately  included  in  the  denominator 
and  received  the  required  numerator  service(s).  In  cases  where  there  are  fewer  than  30  numerator 
positives,  the  EQRO  will  review  all  records  for  that  measure. 

To  provide  sufficient  time  for  each  MCO/PHP  to  gather  the  required  medical  record 
documentation,  the  EQRO  may  direct  the  MCOs/PHPs  to  submit  their  lists  of  members  in  their 
hybrid  sample  twice  -  the  first  list  as  a  preliminary  submission  and  the  second  list  as  a  final 
submission.  Submitting  a  first  list  prior  to  completion  of  the  medical  record  review  process 
would  allow  an  MCO/PHP  additional  time  to  retrieve  medical  record  documentation.  Soon  after 
receipt  of  the  first  list,  the  EQRO  will  provide  the  MCO/PHP  with  the  list  of  medical  records  for 
which  documentation  must  be  submitted.  Only  a  portion  of  the  30  medical  records  for  the 
validation  sample  will  be  included  in  the  EQRO's  first  sample  request  list.  The  remainder  of  the 
30  records  will  be  selected  from  the  final  list.  While  the  first  submission  of  medical  record 
review  findings  is  optional,  it  is  recommended. 

The  EQRO  would  accept  the  first  list  submission  approximately  one  month  prior  to  the 
scheduled  audit.  If  an  MCO/PHP  chooses  to  submit  a  first  list  of  medical  records,  it  must  still 
submit  a  final  listing  sufficiently  in  advance  of  the  scheduled  audit  as  directed  by  the  EQRO.  For 
each  submission,  MCOs/PHPs  will  need  to  identify  all  members  for  whom  medical  record 
review  has  been  conducted  and  indicate  which  members  have  been  found  to  be  numerator 
positives  through  medical  record  review.  The  final  list  must  reflect  the  MCO's/PHP's  final 
medical  record  review  findings,  with  members  for  whom  a  medical  record  was  never  found 
identified  as  not  having  met  the  numerator  requirements. 
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No  predetermined  "passing"  grade  will  be  set  for  the  medical  record  audit.  Rather,  onsite 
auditors  will  use  the  medical  record  review  results  to  determine  if  the  hybrid  rate  or  solely 
medical  record  review  rate,  as  a  whole,  is  biased,  and  to  what  extent  that  bias  affects  the  final 
reported  rate  for  that  measure.  The  EQRO  will  identify  to  the  State  what  effects  bias,  as  well  as 
incomplete  data,  will  have  on  the  MCO's/PHP's  calculation  of  the  performance  measure.  For 
each  of  the  evaluated  measures  auditors  will  determine  the  impact  of  the  findings  from  the  MRR 
validation  process  on  the  MCO's/PHP's  Final  Audit  Designation. 

Step  1:  Calculation  of  the  Medical  Record  Review  Error  Rate 

The  EQRO  will  review  up  to  30  records  identified  by  the  MCO/PHP  as  meeting  numerator 
requirements  (as  determined  through  medical  record  review)  for  the  measures  audited.  Records 
are  randomly  selected  from  the  entire  population  of  MRR  numerator  positives  identified  by  the 
plan,  as  indicated  on  the  MRR  numerator  listings  submitted  to  the  EQRO.  If  fewer  than  30 
medical  records  are  found  to  meet  numerator  requirements,  all  records  are  reviewed. 
Administrative  numerator  positives  are  not  included  as  part  of  this  validation  process.  The 
EQRO  will  calculate  a  medical  record  review  error  rate  for  each  performance  measure  calculated 
by  the  hybrid  method  or  solely  from  medical  record  review  as  illustrated  in  TABLE  4,  below: 


TABLE  4:  Summary  of  Medical  Record  Review  (MRR)  Reabstraction  Findings: 


Column  A 

Column  B 

Column  C 

Column  D 

Column  E 

Column  F 

Performance 
Measure 

Number  of  MMR 
Positives  Selected 
for  Audit 

Number  of 
Medical  Records 
Received 

Number  of  Medical 
Records  Found  to 
be  Compliant 

Accuracy  Rate 
(%)  (D/B) 

Error  Rate  (%) 
(100% -E) 

Column  A:     Name  of  performance  measure  evaluated. 

Column  B:      Total  number  of  MRR  numerator  positive  records  reabstracted  by  EQRO  as  part 
of  the  medical  record  review  validation  process  (i.e.,  30,  or  the  total  population,  if 
less  than  30  MRR  numerator  positives  were  reported). 

Column  C:      Total  number  of  medical  records  submitted  to  EQRO,  as  part  of  the  medical 

record  review  validation  process  (i.e.,  should  be  equal  to  Column  B  or  less  than 
Column  B  if  one  or  more  records  were  not  submitted  on  time). 

Column  D:     Total  number  of  medical  records  reviewed  by  EQRO  and  identified  as  meeting 
numerator  requirements. 

Column  E:      Accuracy  rate  -  percent  of  records  selected  for  audit  that  were  identified  as 
meeting  numerator  requirements  (Column  D/Column  B). 
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Column  F:      Error  rate  -  percent  of  records  selected  for  audit  that  were  identified  as  not 
meeting  numerator  requirements  (100%  -  Column  E). 

Step  2:  Determining  the  Potential  Impact  of  MRR  Reabstraction  Findings  On  Final  Audit 
Designations 

The  next  step  in  MRR  validation  is  to  determine  whether  any  medical  record  review  errors 
significantly  biased  the  final  reported  rate  for  a  given  performance  measure.  To  make  this 
determination,  the  EQRO,  as  directed  by  the  State,  should  develop  and  follow  decision  rules  such 
as  the  following: 

Sample  Decision  Rules: 

Error  Rate  of  10  Percent  or  Less:  If  the  error  rate  (TABLE  4,  column  F)  is  10  percent  or  less,  then 
the  measure  automatically  passes  the  medical  record  review  validation.  The  Final  Audit 
Designation  is  then  determined  based  on  the  auditors'  findings  from  the  Information  Systems 
Capabilities  Assessment  (ISCA)  conducted  as  Pre-Onsite  activity  3  and  Onsite  Activity  1 .  As 
long  as  no  errors  leading  to  significant  bias  are  discovered  during  the  other  components  of  the 
audit  process,  the  final  rate  is  considered  as  having  met  the  validation  standards. 

Error  Rate  of  Greater  than  10  Percent:  If  the  error  rate  (TABLE  4,  column  F)  is  greater  than  10 
percent,  then  the  auditors  determine  the  impact  of  the  MRR  validation  findings  on  the  final 
reported  rate  for  the  measure.  For  each  of  the  measures  under  review,  auditors  evaluate  the 
impact  of  the  plan's  MRR  processes  on  its  final  reported  rate  by  extrapolating  the  findings  from 
the  audited  medical  record  sample  to  the  universe  of  all  MRR  positives.  Details  on  this  process 
are  provided  in  TABLE  5. 

The  maximum  amount  of  bias  allowed  for  the  final  rate  to  be  considered  reportable  is  "x" 
percentage  points  (to  be  determined  by  each  State). 

•  If  the  amount  of  error  in  the  MCO's/PHP's  medical  record  review  process  (TABLE  5,  line  8) 
does  not  cause  the  final  reported  rate  to  be  biased  by  more  than  x  percentage  points,  then  the 
measure  passes  the  medical  record  review  validation.  The  compliance  designation  is  then 
determined  based  solely  on  the  auditors'  findings  from  the  Information  Systems  Capabilities 
Assessment  (ISCA).  As  long  as  no  errors  leading  to  significant  bias  are  discovered  during  the 
other  components  of  the  performance  measure  audit  process,  the  final  rate  is  considered 
valid. 

•  If  the  amount  of  error  in  the  MCO's/PHP's  medical  review  process  (TABLE  5,  line  8) 
ultimately  causes  the  final  reported  rate  to  be  biased  by  more  than  x  percentage  points,  the 
rate  is  automatically  considered  invalid.  The  performance  measure  is  then  designated  as 
invalid. 
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TABLE  5:  Impact  of  MRR  Findings 


Line 

# 

Description 

Measure  A 

Measure  B 

Measure  C 

1 

Final  Data  Collection  Method  Used 
(e.g.,  medical  record  review,  hybrid,) 

2 

Error  Rate  (Percentage  of  records 
selected  for  audit  that  were  identified  as 
not  meeting  numerator  requirements,  as 
shown  in  TABLE  4,  column  F) 

3 

Is  error  rate  <  10%?  (Yes  or  No) 

-If  yes,  MCO/PHP  passes  MRR 
validation;  no  further  MRR  calculations 
are  necessary. 

—If  no,  the  rest  of  the  spreadsheet  will 
be  completed  to  determine  the  impact 
on  the  final  rate. 

4 

Denominator 

(The  total  number  of  members 
identified  for  the  denominator  of  this 
measure,  as  identified  by  the 
MCO/PHP) 

5 

Weight  of  Each  Medical  Record 

(Impact  of  each  medical  record  on  the 
final  overall  rate;  determined  by 
dividing  100%  by  the  denominator  in 
line  4.) 

6 

Total  Number  of  MRR  Numerator 
Positives  identified  by  the  MCO/PHP 
using  MRR. 

7 

Expected  Number  of  False  Positives 

(Estimated  number  of  medical  records 
inappropriately  counted  as  numerator 
positives;  determined  by  multiplying  the 
Error  Rate  in  line  2  by  line  6,  the  total 
number  of  MRR  numerator  positives 
reported.) 

8 

Estimated  Bias  in  Final  Rate 

(The  amount  of  bias  caused  by  medical 
record  review,  measured  in  percentage 
points;  determined  by  multiplying  the 
Expected  Number  of  False  Positives  in 
line  7  by  line  5,  the  Weight  of  Each 
Medical  Record.) 

If  line  8  is  <x%,  then  the  final  rate  is  not  considered  to  be  significantly  biased  by  medical  record  review  alone.  If  the 
other  components  of  the  audit  process  did  not  identify  any  other  issues  that  would  introduce  bias  into  the  rate,  the 
rate  will  be  considered  valid. 


If  line  8  is  >x%,  then  the  final  rate  is  considered  to  be  significantly  biased.  The  measure  will  be  considered  invalid. 
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ATTACHMENT  XIII 


Numerator  Validation  Findings  -  Reviewer  Worksheet 


Audit  Element 

Met 

Not  Met 

N/A 

Comments 

All  appropriate  data  are  used  to  identify  the  entire  at-risk  population. 

The  MCO/PHP  has  used  the 
appropriate  data,  including  linked  data 
from  separate  data  sets,  to  identify  the 
entire  at-risk  population. 

The  MCO/PHP  has  in  place  and 
utilizes  procedures  to  capture  data  for 
those  performance  indicators  that  could 
be  easily  under-reported  due  to  the 
availability  of  services  outside  the 
MCO/PHP. 

Qualifying  medical  events  (such  as  diagnoses,  procedures,  prescriptions,  etc.)  are  properly 
identified  and  confirmed  for  inclusion  in  terms  of  time  and  services 

The  MCO's/PHP's  use  of  codes  used 
to  identify  medical  events  are 
complete,  accurate,  and  specific  in 
correctly  describing  what  has 
transpired  and  when. 

The  MCO/PHP  correctly  evaluated 
medical  event  codes  when  classifying 
members  for  inclusion  or  exclusion  in 
the  numerator. 

The  MCO/PHP  has  avoided  or 
eliminated  all  double-counted  members 
or  numerator  events. 

Any  non-standard  codes  used  in 
determining  the  numerator  have  been 
mapped  to  a  standard  coding  scheme  in 
a  manner  that  is  consistent,  complete, 
and  reproducible  as  evidenced  by  a 
review  of  the  programming  logic  or  a 
demonstration  of  the  program. 

Any  time  parameters  required  by  the 
specifications  of  the  performance 
measure  are  adhered  to  (i.e.,  that  the 
measured  event  occurred  during  the 
time  period  specified  or  defined  in  the 
performance  measure). 
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Audit  Element 

Met 

Not  Met 

N/A 

Comments 

Medical  record  data  extracted  for  inclusion  in  the  numerator  are  properly  collected 

Medical  record  reviews  and 
abstractions  have  been  carried  out  in  a 
manner  that  facilitates  the  collection  of 
complete,  accurate,  and  valid  data. 

•     Record  review  staff  have  been 
properly  trained  and  supervised  for 
the  task. 

•     Record  abstraction  tools  require 
the  appropriate  notation  that  the 
measured  event  occurred. 

•     Record  abstraction  tools  require 
notation  of  the  results  or  findings 
of  the  measured  event  (if 
applicable). 

•     Data  included  in  the  record  extract 
files  are  consistent  with  data  found 
in  the  medical  records  as 
evidenced  by  a  review  of  a  sample 
of  medical  record  for  applicable 
performance  measures.  (From 
Medical  Record  Review 
Validation  Tools-Table  5, 
ATTACHMENT  XII) 

The  process  of  integrating 
administrative  data  and  medical  record 
data  for  the  purpose  of  determining  the 
numerator  is  consistent  and  valid. 
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ATTACHMENT  XIV 


Policies,  Procedures,  Data,  and  Information  Used  to  Implement  Sampling: 

Review  Worksheet 


Documents 

Reviewed 

Not  Reviewed 

Comments 

Description  of  software  used  to  execute 
sampling  sort  of  population  files  when 
sampling  (systematic)  is  used. 

Policies  to  maintain  files  from  which  the 
samples  are  drawn  in  order  to  keep 
population  intact  in  the  event  that  a 
sample  must  be  re-drawn,  or 
replacements  made. 

Computer  source  code  or  logic 
identifying  specified  sampling 
techniques,  and  documentation  that  the 
logic  matches  the  specifications  set  forth 
for  each  performance  measure,  including 
sample  size  and  exclusion  methodology. 

Methods  used  for  sampling  for  measures 
calling  for  hybrid  data  or  medical  record 
review. 

Documentation  assuring  that  sampling 
methodology  treats  all  measures 
independently,  and  that  there  is  no 
correlation  between  drawn  samples. 

Observation  of  or  documentation  of 
procedures  in  which  a  biased  sample  was 
identified  and  corrected. 

Documentation  of  "frozen"  or  archived 
files  from  which  the  samples  were 
drawn,  and  if  applicable,  documentation 
of  the  MCO's/PHP's  process  to  re-draw 
a  sample  or  obtain  necessary 
replacements. 

Describe  Documentation  Review  and  Demonstrations  Provided: 
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ATTACHMENT  XV 


Sampling  Validation  Findings  -  Reviewer  Worksheet 


Audit  Element 

Met 

Not  Met 

N/A 

Comments 

The  MCO/PHP  has  followed  the  specified  sampling  method  to  produce  an  unbiased  sample 
which  is  representative  of  the  entire  at-risk  population. 

•     Each  relevant  member  or  provider 
had  an  equal  chance  of  being 
selected;  no  one  was 
systematically  excluded  from  the 
sampling. 

•     The  MCO  /  PHP  followed  the 
specifications  set  forth  in  the 
penormance  measure  regaruing 
the  treatment  of  sample  exclusions 
and  replacements,  and  if  any 
activity  took  place  involving 
replacements  of  or  exclusions  from 
the  samnle  the  MCO/PHP  kenf 
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adequate  documentation  of  that 
activity. 

•     Each  provider  serving  a  given 
number  of  enrollees  had  the  same 
probability  of  being  selected  as 
any  other  provider  serving  the 
same  number  of  enrollees. 

•     The  MCO/PHP  examined  its 
sampled  files  for  bias,  and  if  any 
bias  was  detected,  the  MCO/PHP 
is  able  to  provide  documentation 
that  describes  any  efforts  taken  to 
correct  it. 

•     The  sampling  methodology 
employed  treated  all  measures 
independently,  and  there  is  no 
correlation  between  drawn 
samples. 

•     Relevant  members  or  providers 
who  were  not  included  in  the 
sample  for  the  baseline 
measurement  had  the  same  chance 
of  being  selected  for  the  follow-up 
measurement  as  providers  who 
were  included  in  the  baseline. 
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Audit  Element 

Met 

Not  Met 

N/A 

Comments 

The  MCO/PHP  maintains  its  performance  measurement  population  files/  data  sets  in  a 
manner  which  allows  a  sample  to  be  re-drawn,  or  used  as  a  source  for  replacement 

•     The  MCO/PHP  has  policies  and 
procedures  to  maintain  files  from 
which  the  samples  are  drawn  in 
order  to  keep  the  population  intact 
in  the  event  that  a  sample  must  be 
re-drawn,  or  replacements  made, 
and  documentation  that  the 
original  population  is  intact. 

Sample  sizes  collected  conform  to  the  methodology  set  forth  in  the  performance  measure 
specifications,  and  the  sample  is  representative  of  the  entire  population. 

•     Sample  sizes  meet  the 

requirements  of  the  performance 
measure  specifications. 

•     The  MCO/PHP  has  appropriately 
handled  the  documentation  and 
reporting  of  the  measure  if  the 
requested  sample  size  exceeds  the 
population  size. 

•     The  MCO/PHP  properly 
oversampled  in  order  to 
accommodate  potential  exclusions. 

For  performance  measures  which  include  medical  record  reviews  (e.g.,  hybrid  data 
collections  methodology),  proper  substitution  methodology  was  followed 

•     Substitution  applied  only  to  those 
members  who  met  the  exclusion 
criteria  specified  in  the 
performance  measure  definitions 
or  requirements. 

•     Substitutions  were  made  for 

properly  excluded  records  and  the 
percentage  of  substituted  records 
was  documented. 
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Using  Performance  Measures 


Using  Performance  Measures 

Michael  H.  Bailit 
Bailit  Health  Purchasing,  LLC 
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Michael  H.  Bailit,  M.B.A. 


Michael  is  the  president  of  Bailit  Health  Purchasing,  LLC  (BHP),  a  Massachusetts-based  firm 
that  assists  public  agencies  and  purchaser  coalitions  in  the  purchase  of  services  from,  and  the 
management  of  performance  by,  managed  care  plans  and  service  providers.  BHP  also  assists 
states  in  new  program  design  and  policy  development.  Michael  has  assisted  the  states  of 
Connecticut,  Georgia,  Minnesota  and  Texas  in  new  program  development.  He  has  provided 
managed  care  procurement,  performance  monitoring,  and  evaluation  services  for  the  states  of 
Iowa,  Massachusetts,  New  Hampshire,  New  Jersey,  New  York,  Pennsylvania,  Vermont,  and 
Virginia,  and  has  trained  state  staff  on  managed  care  vendor  oversight  in  Maryland,  Michigan, 
New  Mexico,  and  Pennsylvania. 

Prior  to  forming  BHP,  Michael  served  as  the  Assistant  Commissioner  for  Benefit  Plans  in  the 
Massachusetts  Division  of  Medical  Assistance,  the  state  Medicaid  agency.  His  responsibilities 
included  the  management  of  all  of  the  Division's  benefit  plans,  including  the  HMO,  behavioral 
health,  primary  care  case  management,  and  senior  care  programs.  He  also  implemented  the 
Massachusetts  enrollment  broker  program,  which  was  the  first  such  statewide  program  in  the 
country.  For  each  of  these  programs  he  designed  and  supervised  the  execution  of 
procurements  as  well  as  ongoing  vendor  performance  management.  He  was  also  responsible 
for  new  managed  care  program  development  for  the  Division,  including  the  Division's  HCFA 
1115  health  reform  waiver  and  its  1115  Medicare-Medicaid  dual-eligibles  waiver.  While  with 
Massachusetts,  he  served  as  chair  of  the  HCFA  Medicaid  Managed  Care  Technical  Advisory 
Group. 

Also  while  with  Massachusetts,  Michael  founded  the  Massachusetts  Healthcare  Purchaser 
Group  and  served  as  its  chairman  and  president  from  1993-1996.  The  MHPG  is  a  statewide 
coalition  of  67  public  and  private  purchasers  of  managed  care  and  is  the  leading  voice  of 
purchasers  in  the  Commonwealth. 

Previously,  Michael  worked  for  Digital  Equipment  Corporation  and  was  involved  in  planning 
and  management  activities  for  Digital's  U.S.  health  and  welfare  benefits.  Michael  was  heavily 
involved  in  the  development  of  Digital's  innovative  managed  care  program.  While  at  Digital 
he  developed  Digital's  nationally-recognized  HMO  Standards  and  created  their  HMO  network 
management  program.  Prior  to  Digital,  Michael  held  positions  in  the  managed  care  industry. 

Michael  received  his  Bachelor's  degree  at  Wesleyan  University  and  his  M.B.A.  from  the  Kellogg 
Graduate  School  of  Management  at  Northwestern  University. 
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MONICA  C.  WASSELL,  R.N.,  M.P.P. 


Program  Director 

Summary: 

Ms.  Wassell  has  over  12  years  of  experience  with  medical  information  systems  and  over  20  years 
of  experience  in  the  healthcare  field.  She  has  supported  HCFA  External  Quality  Review 
Organization  (EQRO)  protocol  development  and  has  worked  directly  for  the  National  Committee 
of  Quality  Assurance  (NCQA)  to  manage  Managed  Care  Organization  (MCO)  accreditation 
surveys  and  design  improvements  to  NCQA  standards.  Ms.  Wassell  is  responsible  for 
structuring,  implementing,  and  managing  project  work  plans,  including  facilitating 
multidisciplinary  team  meetings;  assuring  that  budgetary  and  time  constraints  are  properly 
managed;  and  preparing  interim  and  final  project  reports  that  meet  client  specifications.  She 
manages  direct  project  staff,  including  matrixed  resources  in  finance,  contract,  and  technical 
divisions  of  the  company.  Ms.  Wassell  assists  in  developing  technical  proposals  for  federal, 
state,  and  commercial  projects.  In  her  current  role,  Ms.  Wassell  is  responsible  for  contracts  that 
support  the  National  Quality  Management  Program  for  the  Department  of  Defense,  and  the  State 
of  Texas,  for  its  external  quality  monitor  for  Medicaid  managed  care,  the  Texas  Health  Quality 
Alliance  (THQA),  with  tasks  that  include  the  collection,  analysis,  and  reporting  of  performance 
and  outcome  measures  for  hospital  accreditation,  HCFA  requirements  for  Medicare  managed 
care  plans,  and  Medicaid  Managed  Care  evaluation;  special  studies  focused  on  compliance  with 
clinical  practice  guidelines  and  other  disease  management  activities;  as  well  as  training  and 
education  programs  to  assist  clients  in  data-driven  quality  improvement  activities  associated  with 
health  care  delivery. 

Education: 

1995,  MPP,  Public  Policy,  Duke  University 

1986,  BA,  Economics,  University  of  California  at  Berkeley 

1971,  RN,  Nursing,  Evanston  Hospital  School  of  Nursing 


FMAS  Corporation,  DynCorp  Company 


Monica  C  Wassell,  Rev.  6/19/00 
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BIOGRAPHY 
Barry  S.  Kast,  MSW 


Mr.  Kast  is  the  Administrator  of  the  Oregon  Mental  Health  and  Developmental 
Disability  Services  Division.  He  has  worked  in  public  mental  health  services 
for  more  than  twenty  years  with  experience  in  adult  day  treatment,  acute 
inpatient  care,  as  director  of  a  county  mental  health  center  and  as  state 
mental  health  program  director.  At  the  national  level,  Mr.  Kast  has  served  on 
the  Board  of  Directors  for  the  National  Association  of  State  Mental  Health 
Program  Directors  (NASMHPD)  from  1994  to  the  present,  and  is  the  current 
Vice  President  of  NASMHPD  (2000  -  2001 ). 

He  has  been  actively  involved  in  the  management  of  system  change  in 
Oregon  including  state  hospital  reduction,  capitated  children's  programming, 
state-level  planning  for  Oregon's  incorporation  of  mental  health  services  in 
health  reform,  and  the  relocation  of  training  center  residents  (ICF/MR)  to 
community  settings. 

Mr.  Kast  is  a  graduate  of  Stanford  University  with  advanced  degrees  in 
English  from  the  State  University  of  New  York  and  social  work  from  Portland 
State  University.  He  resides  with  his  wife,  son  and  daughter  in  Corvallis, 
Oregon. 
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Performance  Measures: 
Behavioral  Health 

Deborah  Garnick 
Brandeis  University 


Deborah  Garnick 


Deborah  Garnick,  Sc.D.,  is  a  Research  Professor  at  the  Institute  for  Health  Policy  at  the  Heller 
School  at  Brandies  University.  She  reached  her  doctorate  in  health  services  research  from  the  School 
of  Hygiene  and  Public  Health  at  Johns  Hopkins  University.  In  1997  she  was  elected  a  Fellow  of  the 
Association  for  Health  Services  Research. 

Before  coming  to  Brandies  in  1992,  she  was: 

•  Deputy  Director  of  the  Program  on  Health  Care  Financing  and  Insurance  in  the  Department 
of  Health  Policy  and  Management  of  the  Harvard  School  of  Public  Health  (1989-1992) 

•  Research  Associate  at  the  Institute  for  Health  Policy  at  the  University  of  California,  San 
Francisco  (1983-1989) 

A  Service  Fellow  at  the  National  Center  for  Health  Services  Research  of  the  federal 
Department  of  Health  and  Human  Services  during  her  graduate  training  at  Johns  Hopkins 

Currently,  Dr.  Garnick  is  focusing  on  two  research  projects.  First,  she  is  the  Co-Principal 
Investigator  on  studies  that  involve  tracking  of  drug  and  alcohol  abuse  services  and  mental  health 
services  provided  by  the  managed  behavioral  health  care  industry  (funded  by  the  National  Institute 
on  Drug  Abuse,  National  Institute  on  Alcoholism  and  Alcohol  Abuse  and  the  Substances  Abuse  and 
Mental  Health  Administration).  Second,  she  is  directing  an  effort  to  develop  performance  measures 
for  substance  abuse  services  in  managed  care  plans. 

Dr.  Garnick  has  published  widely  on  the  topics  of  state-level  health  insurance  reform  legislation, 
quality  of  hospital  and  ambulatory  care,  and  methods  of  using  data  collected  for  administrative 
purposes  for  health  services  research. 
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Performance  Measures: 
Behavioral  Health 

Candice  Nardini 
Value  Options 


Candice  Woods  Nardini 


Candice  Woods  Nardini  (Candy)  has  more  than  28  years  of  experience  in  mental  health 
service  delivery  systems.  She  has  worked  in  both  institutional  and  provider  agencies  in 
Iowa,  and  was  program  director  for  Iowa's  Mental  Health  Access  Plan  as  well  as  the 
state's  second  generation  of  managed  behavioral  health  care,  the  Iowa  Plan  for 
Behavioral  Health.  When  she  left  the  Iowa  Department  of  Human  Services,  Candy  was 
Chief  of  the  Bureau  of  Health  Care  Purchasing  and  Quality  Management  with  Iowa's 
Medicaid  program  and  had  responsibility  for  all  $1.8  billion  in  Medicaid-funded  services 
in  Iowa,  including  the  state's  managed  physical  and  behavioral  health  care  programs. 
Since  June  of  1999,  Candy  has  been  vice  president  for  public  sector  development  with 
ValueOptions  and  works  in  the  design  and  implementation  of  the  company's  public 
sector  managed  behavioral  health  care  programs.  She  has  been  active  in 
ValueOptions'  efforts  in  Nebraska,  Pennsylvania,  New  Jersey,  Connecticut  and  other 
states  and  counties  across  the  nation. 

Candy  has  a  bachelor's  degree  from  the  University  of  Missouri  and  a  master's  degree  in 
public  administration  from  the  University  of  Nebraska. 


Performance  Measures: 
Acute/Preventive  Health 


I 


Performance  Measures: 
Acute/Preventive  Health 

Mary  Fermazin 
Health  Services  Advisory  Group 


I 

I 


Mary  Fermazin 


Dr.  Mary  Fermazin  retired  from  the  California  Department  of  Health  Services  in 
August,  2000.  During  her  12  year  tenure  with  the  California  Medicaid  Program, 
(Medi-Cal),  she  served  in  various  roles  including:  establishing  medical  policies 
and  utilization  review  criteria,  evaluating  new  benefits  and  technology  for  the 
Medi-Cal  Fee  For  Service  program.  As  the  Chief  of  the  Office  of  Clinical 
Standards  and  Quality  in  the  Medi-Cal  Managed  Care  program,  she  led  the  effort 
to  establish  a  distinctive  set  of  clinical  standards  and  a  cutting  edge  quality 
improvement  program.  Over  the  years,  she  had  participated  in  various  quality 
forums,  both  at  the  national  and  state  level,  including  the  NCQA  Committee  for 
Performance  Measurement  for  the  development  of  the  Health  Plan  Employer 
Data  Set  (HEDIS®),  and  the  HCFA  Quality  Assurance  Technical  Advisory  Group. 
She  enjoyed  an  exceptional  reputation  for  her  excellent  relationship  with  health 
plans,  advocates,  provider  associations  and  public  organizations.  Dr.  Fermazin 
is  now  the  Vice  President  of  Medical  Affairs  of  Health  Services  Advisory  Group, 
Inc.  (HSAG),  based  in  Phoenix  Arizona.  HSAG  is  a  quality  improvement 
organization  serving  national  clients  in  areas  such  as  external  quality  review, 
health  outcomes  measurement,  surveys,  and  HEDIS®  audits  for  national  clients. 


Note:  HEDIS  is  a  registered  trademark  of  the  National  Committee  for  Quality  Assurance  (NCQA). 
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Performance  Measures: 
Acute/Preventive  Health 

Foster  C.  Gesten 
New  York  Department  of  Health 
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Foster  Gesten,  MD 


Foster  Gesten  is  a  board  certified  internist  who  is  currently  the  Medical  Director  for  the 
Office  of  Managed  Care  (OMC)  for  the  New  York  State  (NYS)  Department  of  Health. 
The  Office  of  Managed  Care  provides  oversight  for  all  managed  care  plans  in  NYS  and 
includes  responsibility  for  the  management  of  the  Medicaid  Managed  Care  Program.  His 
major  responsibilities  include  clinical  management  of  programs  in  quality  measurement 
and  improvement,  development  of  clinical  standards  of  care,  program  evaluation  and 
outcomes  research.  Prior  to  becoming  Medical  Director  for  OMC,  he  was  a  Senior 
Medical  Research  Scientist  in  the  Office  of  Quality  Improvement  at  the  Department  of 
Health. 

Before  joining  the  Health  Department  in  1994,  Dr.  Gesten  was  an  internist  in  a  staff 
model  HMO  in  Albany,  New  York.  He  received  his  training  in  general  internal  medicine 
at  Brown  University,  Rhode  Island  Hospital. 


Performance  Measures: 
Acute/Preventive  Health 

Alice  Lind 

Washington  Medical  Assistance  Administration 


Alice  Lind 

Alice  Lind  currently  holds  the  position  of  Manger,  Quality  Management  Section, 
Department  of  Social  and  Health  Services,  Washington  State  Medical  Assistance 
Administration.  She  has  worked  in  Quality  Improvement  since  1993,  first  with  private 
sector,  home  health  agencies,  and  since  1995,  for  the  Washington  State  Medicaid  agency. 
Alice's  previous  experience  includes:  direct  patient  care  in  both  Intensive  Care  and 
Hospice,  and  as  a  nursing  manager  of  a  long  term  care  facility  for  persons  with  AIDS. 
Alice  received  her  RN  and  her  MPH  from  the  University  of  North  Carolina. 
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Performance  Measures: 
Special  Needs  Populations 
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Performance  Measures: 
Special  Needs  Populations 

Susan  Moran 
Michigan  Department  of  Community  Health 


Susan  Moran 


Sue  Moran  is  Manager  of  the  Quality  System  Section  within  the  Quality 
Improvement  and  Customer  Service  Bureau  of  Michigan's  Medical  Services 
Administration  (the  State's  Medicaid  agency).  Her  section  is  responsible  for 
many  of  the  Medicaid  program  quality  oversight  initiatives,  including  the  external 
quality  review,  HEDIS  analysis,  consumer  satisfaction  survey,  consumer  guide, 
and  encounter  data  reporting.  Sue  has  been  employed  with  the  Michigan 
Department  of  Community  Health  since  September  1998.  Prior  to  joining  DCH, 
she  worked  for  over  ten  years  in  the  managed  care  industry,  primarily  in  quality 
assessment  and  improvement.  Sue's  background  is  in  nursing  and  public  health. 

She  will  discuss  with  us  the  work  Michigan  has  done  in  the  development  and 
implementation  of  a  structured  site  visit  tool  modeled  after  their  Medicaid 
contract  and  used  by  the  state  to  monitor  compliance  with  structural  and 
operational  standards. 


Performance  Measures: 
Special  Needs  Populations 

Elaine  Haemmerle 
Arizona  Health  Care  Cost  Containment  System 


Elaine  Haemmerle 


Elaine  Haemmerle  is  a  member  of  the  Office  of  Medical  Management(OMM)  Research  Unit  for 
Arizona  Health  Care  Cost  Containment  System  (AHCCCS),  Arizona's  Medicaid  agency.  Elaine 
has  worked  for  AHCCCS  since  leaving  the  Air  Force  in  1994,  focusing  primarily  on  Long  Term 
Care.  She  has  worked  in  various  aspects  of  nursing  since  graduating  from  Arizona  State 
University  in  1978.  Areas  of  experience  include:  Military  Medicine,  Intensive  Care, 
Medical/Surgical,  Pediatric,  Home  Health,  Managed  Care,  Emergency  Medicine, 
Utilization/Quality  Management,  Urgent  Care  and  Research.  During  her  career  she  has 
designed,  developed,  and  implemented  a  variety  of  health  care  related  programs  such  as  urgent 
care  clinics,  air  show  emergency  coverage,  CQI,  and  quality  management.  In  addition  she 
functions  as  a  consultant  on  database  development  and  training.  Currently  she  has  been 
functioning  as  the  Long  Term  Care  Research  Coordinator  for  OMM  with  responsibility  for  the 
Long  Term  Care  Quality  Indicators,  data  processing  and  analysis,  database  development,  and 
data  sharing. 
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VALIDATING  PERFORMANCE  IMPROVEMENT  PROJECTS 

(IN  MANAGED  CARE  ORGANIZATIONS  AND  PREPAID  HEALTH  PLANS) 

I.  PURPOSE  OF  THE  PROTOCOL 

The  purpose  of  health  care  quality  performance  improvement  projects  (PIPs)  is  to  assess  and 
improve  processes,  and  thereby  outcomes,  of  care.  In  order  for  such  projects  to  achieve  real 
improvements  in  care,  and  for  interested  parties  to  have  confidence  in  the  reported 
improvements,  PIPs  must  be  designed,  conducted  and  reported  in  a  methodologically  sound 
manner.  This  protocol  specifies  procedures  for  external  quality  review  organizations  (EQROs)1 
to  use  in  evaluating  the  methodologic  soundness  and  reported  successes  of  PIPs  implemented  by 
Medicaid  Managed  Care  Organizations  (MCOs)  and  Prepaid  Health  Plans  (PHPs). 

II.  ORIGIN  OF  THE  PROTOCOL 

This  protocol  was  developed  from  existing  standards  and  guidelines  that  are  used  in  the  public 
and  private  sectors.  These  include: 

•  Quality  Improvement  System  for  Managed  Care  (QISMC) 

QISMC  is  an  initiative  of  HCFA  that  sets  forth  standards  and  guidelines  pertaining  to 
health  care  quality  for  Medicare  and  Medicaid  health  plans  (MCOs,  PHPs,  and 
Medicare+Choice  plans).  These  standards  and  guidelines,  in  part,  address  MCO  and  PHP 
quality  assessment  and  performance  improvement  (QAPI)  projects.  The  QISMC 
standards  and  guidelines  provided  guidance  to  HCFA  in  the  drafting  of  proposed 
regulations  to  implement  the  quality  requirements  established  by  the  Balanced  Budget 
Act  of  1997  (BBA)  for  Medicaid  MCOs,  including  those  pertaining  to  MCO  and  PHP 
performance  improvement  projects. 

•  Health  Care  Quality  Improvement  Studies  in  Managed  Care  Settings  (HCQIS):  A  Guide 
for  State  Medicaid  Agencies  (National  Committee  for  Quality  Assurance  (NCQA)) 

Produced  under  a  contract  from  HCFA,  this  guidebook  identifies  key  concepts  related  to 
the  conduct  of  quality  improvement  studies  and  details  widely  accepted  principles  of 
research  design  and  statistical  analysis  necessary  for  designing,  implementing  and 
assessing  quality  improvement  studies. 


It  is  recognized  that  a  State  Medicaid  agency  may  choose  an  organization  other  than  an  EQRO  (as  defined 
in  Federal  regulation)  to  validate  MCO  or  PHP  performance  improvement  projects.  However,  for  convenience,  in 
this  protocol  we  use  the  term  "external  quality  review  organization  (EQRO)"  to  refer  to  any  organization  that 
validates  performance  improvement  projects  undertaken  by  MCOs  or  PHPs. 
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•  A  Health  Care  Quality  Improvement  System  for  the  Medicaid  Managed  Care  (QARI),  A 
Guide  for  States  (Health  Care  Financing  Administrations  (HCFA)) 

HCFA's  1993  guide  for  health  care  quality  improvement  provides  a  framework  for 
building  (QI)  systems  within  State  Medicaid  managed  care  initiatives.  This  document 
included  guidelines  addressing  quality  assessment  and  improvement  studies  and  related 
activities  of  MCOs  and  PHPs. 

•  Framework  for  Improving  Performance,  From  Principles  to  Practice  (Joint  Commission 
on  Accreditation  of  Healthcare  Organizations  (JCAHO) 

This  publication  describes  the  Joint  Commission's  theory-based,  practical  methodology 
for  continuously  improving  the  core  work  and  resulting  outcomes  of  any  health  care 
organization.  In  this  document,  JCAHO  defines  the  key  characteristics  and  essential 
behaviors  of  any  health  care  organization  striving  to  achieve  high  quality  patient  care. 

1 990-2000  Standards  for  Health  Care  Networks  (SHCN)  (JCAHO) 

The  JCAHO  1990-2000  SHCN  provides  a  standards-based  evaluation  process  to  assist 
the  MCO  in  measuring,  assessing  and  improving  its  network's  performance.  It  also  helps 
the  MCO  focus  on  conducting  performance  improvement  efforts  in  a  multi  disciplinary, 
system-wide  manner.  The  1 990-2000  SHCN  integrates  information  about  the  Joint 
Commission's  health  care  network  accreditation  process. 

•  NCQA  1997,  1998,  and  1999  Standards  for  Accreditation  of  Managed  Care 
Organizations  and  NCQA  1999  Standards  for  Accreditation  of  Managed  Behavioral 
Healthcare  Organizations  (MBHO) 

These  documents  include  administrative  policies  and  procedures  for  NCQA's  MCO  and 
MBHO  accreditation  programs,  the  1997,  1998,  and  1999  standards,  and  rationale 
statements  for  the  standards. 

Peer  Review  Organizations  (PRO)  4th  and  5th  Scope  of  Work  (SO  W)  (HCFA) 

The  4th  and  5th  SOW  documents  outline  the  requirements  for  PROs  to  adhere  to  while 
conducting  health  care  quality  and  improvement  activities  for  Medicare  beneficiaries. 

An  in-depth  comparison  of  these  documents  was  performed  to  identify  the  activities  and  features 
common  to  these  protocols,  and  features  unique  to  individual  protocols,  while  acknowledging  the 
different  purposes  of  the  documents.  The  QISMC,  JCAHO  and  NCQA  standards  are  written  as 
guides  for  MCOs/PHPs  to  follow  in  developing,  conducting  and  evaluating  their  performance 
improvement  projects.  They  can  also  be  used  by  States  or  their  agents  (e.g.,  EQROs)  to  assess 
compliance  with  State  mandated  guidelines  and/or  to  facilitate  overall  plan-to-plan  comparisons. 
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QARI  was  written  with  States  as  the  intended  audience  to  help  them  and  their  agents  (e.g., 
EQROs)  assure  compliance  with  regulations  and  Medicaid  program  requirements,  and  promote 
consistency  in  the  manner  in  which  MCOs  and  PHPs  carry  out  activities  related  to  performance 
improvement  projects. 

The  analysis  revealed  that  in  spite  of  their  different  purposes,  all  documents  identify  several 
common  characteristics  of  effective  performance  improvement  projects.  These  include: 

Selection  of  Topics:  All  of  the  reference  documents  address  the  need  for  PIPs  to  clearly  specify 
the  topic  to  be  addressed.  They  all  acknowledge  both  clinical  (e.g.,  specific  disease  or  condition 
such  as  pregnancy  or  asthma)  and  non-clinical  (e.g.,  availability,  timeliness  and  accessibility  of 
care)  health  service  delivery  issues  as  appropriate  topics  for  health  care  quality  improvement 
initiatives. 

Means  of  Identifying  Topics:  Continuous  data  collection  and  analysis  is  stressed  throughout  all 
documents  as  a  means  of  identifying  appropriate  topics.  It  is  stated  that  topics  should  be 
systematically  selected  and  prioritized  to  achieve  the  greatest  practical  benefit  for  enrollees.  A 
minimal  set  of  criteria  is  suggested  for  selecting  the  appropriate  topics,  including:  considering  the 
prevalence  of  a  condition  among,  or  need  for  a  specific  service  by,  an  MCO's/PHP's  enrollees; 
enrollee  demographic  characteristics  and  health  risks,  the  likelihood  that  the  study  topic  will 
result  in  improved  health  status  among  the  enrollees,  and  the  interest  of  consumers  in  the  aspect 
of  care  or  services  to  be  addressed. 

Scope  of  project  topics:  The  QISMC  standards  specify  that  PIPs  should  address  the  breadth  of 
the  MCO's  or  PHP's  services,  such  as  whether  they  include  physical  health  and  mental/substance 
abuse  health  services.  They  also  identify  specific  clinical  and  non-clinical  focus  areas  that  are 
applicable  to  all  enrollees.  The  QISMC  standards  also  specify  that  the  scope  of  the  health  plans' 
improvement  efforts  are  to  include  all  enrollees. 

Stating  the  Study  Question(s):  The  HCQIS  Guide  discusses  the  importance  of  "stating  the  study 
question"  after  a  study  topic  is  identified.  It  emphasizes  that  stating  a  study  question  helps  a 
project  team  avoid  being  sidetracked  by  data  that  is  not  central  to  the  issue  under  study.  For 
example,  once  a  PIP  has  identified  childhood  immunizations  as  a  study  topic:  it  might  specify  a 
number  of  different  study  questions: 

•  Have  all  children  received  all  scheduled  doses  of  one  vaccine  in  particular? 

•  Have  all  children  of  all  ages  received  all  recommended  vaccines  appropriate  for 
their  age? 

•  Have  all  children  of  a  particular  age  (e.g.,  at  the  age  of  one,  two  six  or  other  years) 
received  all  age-appropriate  immunizations? 
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Alternatively,  an  MCO/PHP  might  desire  more  detailed  information,  and  specify  the 
study  questions  as: 

•  What  proportion  of  Medicaid  enrolles  who  have  reached  two  years  of  age  have 
received: 

•  All  four  recommended  doses  of  DPT  vaccine? 

•  All  three  recommended  doses  of  the  Polio  vaccine? 

•  One  recommended  dose  of  the  MMR  vaccine? 

•  At  least  one  dose  of  Hib  in  the  second  year  of  life? 

An  MCO/PHP  might  further  specify  additional  study  questions  to  provide  information  to 
assist  them  in  their  QI  efforts,  such  as: 

•  In  what  percent  of  cases  of  lack  of  immunization  were  children  not  immunized  for 
one  of  the  following  reasons? 

•  Refusal  by  a  parent  or  guardian. 

•  Medical  contraindications. 

•  Member  non-complaint  with  the  recommended  immunization  regimen. 

Incorporating  the  process  of  documenting  a  study  question(s)  into  the  project  design  would  help 
ensure  that  the  MCO/PHP  used  a  systematic  method  of  identifying  appropriate  indicators  and 
data  to  be  collected.  In  this  protocol  we  have  included  "defining  the  study  question(s)"  as  a  key 
step  in  designing  and  implementing  a  PIP. 

Use  of  Quality  Indicators:  All  reference  documents  address  the  need  to  specify  well  defined 
indicators  to  be  monitored  and  evaluated  throughout  the  project.  It  is  emphasized  that  quality 
indicators  do  not  always  need  to  be  outcome  measures.  Process  measures  are  also  appropriate, 
especially  when  there  is  strong  clinical  evidence  that  the  process  being  measured  has  a 
meaningful  association  with  outcomes.  There  are  various  ways  to  obtain  appropriate  indicators, 
such  as  using  those  dictated  from  outside  sources  (such  as  the  State  or  HCFA)  or  by  an 
MCO/PHP  developing  them  internally  on  the  basis  of  clinical  literature  or  findings  of  expert 
panels. 

In  addition  to  these  features  found  uniformly  in  all  reference  documents,  other  significant  aspects 
of  PIPs  were  identified  by  one  or  more  of  the  reference  documents.  These  include: 

Significant  improvement:  HCFA's  Notice  of  Proposed  Rulemaking  (NPRM)  to  implement  the 
BBA  quality  provisions  called  for  MCO/PHP  PIPs  to  demonstrate  "significant"  improvement. 
QISMC  defines  specific  amounts  of  measurable  improvement  to  be  demonstrated  by  the  health 
plan.  "Demonstrable"  improvement  is  defined  as  either:  1)  benchmarks  established  by  HCFA 
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(for  national  Medicare  projects)  or  State  agencies  (for  statewide  Medicaid  QI  projects)  or  by  the 
health  plans  for  individual  (organizational)  projects;  or  2)  a  10%  reduction  in  adverse  outcomes. 

Phase-in  or  time  frame  requirements:  QISMC  delineates  specific  time  frame  requirements  for 
MCOs/PHPs  to  reach  certain  phases  in  a  QI  cycle.  For  example: 

•  By  the  end  of  the  first  year,  an  MCO/PHP  should  have  initiated  at  least  two  PIPs 
addressing  two  different  focus  areas; 

•  By  the  end  of  the  second  review  year,  at  least  two  additional  projects  addressing 
two  different  focus  areas  should  be  initiated. 

•  By  the  end  of  the  first  year  after  the  2  year  phase-in  period,  and  each  subsequent 
year,  at  least  two  projects  are  to  achieve  demonstrable  improvement  in  two  of  the 
focus  areas. 

Evaluation  Tools:  NCQA's  HCQIS  guidebook  includes  study  planning  and  summary 
worksheets  to  be  used  in  the  evaluation  of  an  MCO's/PHP's  PIP.  This  feature  provides  a  helpful 
method  for  recording  data  during  the  evaluation  process  and  promotes  the  collection  of  consistent 
information  by  all  evaluators.  This  protocol  contains  an  example  of  a  monitoring  worksheet  that 
can  be  used  by  reviewers. 

Scoring  system:  NCQA  accreditation  provides  a  numerical  scoring  system  to  measure 
performance  against  standards  and  to  promote  consistency  in  the  process  used  to  evaluate  MCOs. 
Although  the  scores  do  not  dictate  the  final  decision  with  respect  to  compliance  with  standards, 
they  do  serve  as  a  guide  for  NCQA  evaluators  to  recommend  non-compliance.  This  scoring 
system  also  includes  an  opportunity  for  the  MCO/PHP  to  comment  on  the  surveyor's  scores 
before  a  final  decision  is  rendered.  This  allows  the  MCO/PHP  to  correct  any  errors  or  omissions 
before  the  final  decision  is  made  by  the  oversight  committee.  It  also  promotes  continuous 
improvement  practices  by  securing  "customer"  input  into  a  final  product  (i.e.,  evaluation 
decisions).  This  protocol  does  not  include  a  defined  scoring  system  because  compliance 
determinations  will  be  set  by  the  State  agency. 

Overrides:  The  training  provided  to  the  evaluators  is  vital  to  the  success  of  the  scoring  process. 
NCQA  provides  guidelines  to  their  surveyors  to  permit  overriding  the  scoring  of  standards  when 
circumstances  allow  individual  judgement  of  the  survey  team.  As  this  protocol  does  not  include  a 
scoring  process,  an  override  component  is  not  necessary. 

III.      OVERVIEW  OF  THE  PROTOCOL 

This  protocol  incorporates  activities  identified  in  the  preceding  discussion  of  the  existing  public 
and  private  sector  standards/guidelines.  Activities  that  all  standards/guidelines  have  in  common, 
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as  well  as  some  activities  found  in  one  or  more  reference  documents  also  were  included  where 
the  activity  was  felt  to  be  consistent  with  provisions  HCFA  has  included  in  its  NPRM  to 
implement  the  BBA. 

HCFA's  NPRM  implementing  the  BBA  provisions  pertaining  to  PIPs  provided  the  basic 
framework  for  the  protocol.2  The  methodology  was  based  on  that  found  in  the  HCQIS  document. 
The  QISMC  standards  and  guidelines  also  were  a  source  of  much  explanatory  text.  Specific 
features  of  the  NCQA  and  JCAHO  guidelines  were  incorporated  as  appropriate. 

The  protocol  describes  two  major  activities  that  are  to  be  undertaken  in  validating  PIPs:  1) 
assessing  the  MCO's  /PHP's  methodology  for  conducting  the  PIP  and  2)  verifying  actual  PIP 
study  findings.  Activity  one,  Assessing  the  MCO 's  /PHP 's  Methodology  for  Conducting  the  PIP, 
involves  ten  steps: 


1. 

Review  of  the  selected  study  topic(s) 

2. 

Review  of  the  study  question(s) 

3. 

Review  of  the  selected  study  indicator(s) 

4. 

Review  of  the  study  population 

5. 

Review  of  sampling  methods  (if  sampling  was  used) 

6. 

Review  of  data  collection  procedures 

7. 

Assessing  MCO/PHP  improvement  strategies 

8. 

Review  of  MCO/PHP  data  analysis  and  interpretation  of  study  results 

9. 

Assessing  whether  any  documented  improvement  is  "significant" 

10. 

Assessing  whether  the  MCO/PHP  has  sustained  its  documented  improvement 

Assessing  an  MCO's  or  PHP's  methodology  for  conducting  a  PIP  requires  the  EQRO  to  have 
information  on  the  design  and  implementation  of  the  PIP.  This  information  could  be  obtained 
from  a  hardcopy  or  electronic  written  description  of  the  PIP  design  and  implementation  that  is 
transmitted  to  the  State  and/or  the  EQRO.  It  also  could  be  obtained  through  an  interview  of 
MCO/PHP  personnel  responsible  for  the  design  and  conduct  of  the  PIP,  or  a  combination  of  a 
written  description  plus  interviews.  Information  obtained  through  hardcopy  or  electronic 
submission,  or  interview,  may  also  need  to  be  supplemented  by  supporting  documentation 
obtained  from  the  MCO  on  an  ad  hoc  basis.  Whatever  sources(s)  of  information  are  used,  the 
EQRO  should  follow  the  steps  below  to  assess  the  methodology  of  the  MCO's/PHP's  PIP(s). 


2 

When  HCFA  issues  its  final  rule  to  implement  the  BBA,  any  departures  from  the  NPRM  provisions 
addressing  PIPs,  will  be  incorporated  into  this  protocol  as  well. 


IV. 


PROTOCOL  ACTIVITIES 


ACTIVITY  1: 


ASSESS  THE  STUDY  METHODOLOGY 
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Answers  to  the  questions  in  each  of  the  steps  should  be  recorded  on  a  standardized  Project 
Monitoring  Worksheet  such  as  that  located  in  Attachment  1 . 

Because  HCFA's  NPRM  did  not  put  forth  requirements  addressing  how  long  a  MCO/PHP  may 
take  to  complete  the  PIPs  required  by  the  State,  it  is  expected  that  each  State  will  prescribe  these 
requirements.  In  implementing  this  protocol,  the  EQRO  will  need  to  determine  from  the  State: 

1)  whether  the  EQRO  is  to  annually  validate  all  projects  a)  initiated,  b)  underway  but  not 
completed,  and  c)  completed  during  the  reporting  year,  or  d)  some  combination  of  these 
three  stages  of  PIPs. 

2)  whether  the  EQRO  is  to  review  all  projects  in  the  categories  above,  or  a  random 
sample  of  the  PIPs  in  the  particular  categories. 

Step  1:  Review  the  selected  study  topic(s) 

Rationale 

Performance  improvement  projects  for  Medicaid  managed  care  should  address  and  achieve 
improvement  in  relevant  areas  of  clinical  care  and  non-clinical  services.  Topics  selected  for 
study  must  reflect  the  MCO's/PHP's  Medicaid  enrollment  in  terms  of  demographic 
characteristics,  prevalence  of  disease  and  the  potential  consequences  (risks)  of  the  disease.  Note 
that  sometimes  the  State  Medicaid  agencies  may  have  selected  the  MCO's/PHP's  study  topic. 

Potential  Sources  of  Supporting  Information 

•  Data  on  enrollee  characteristics  relevant  to  health  risks  or  utilization  of  clinical  and  non- 
clinical services,  such  as  age,  sex,  race/ethnicity/language  and  disability  or  functional 
status. 

•  Utilization,  diagnostic  and  outcome  information,  such  as  outpatient  and  inpatient 
encounters,  services,  procedures,  medications  and  devices,  admitting  and  encounter 
diagnoses,  adverse  incidents  (such  as  deaths,  avoidable  admissions,  or  readmissions);  and 
patterns  of  referrals  or  authorization  requests. 

•  Data  on  the  MCO's/PHP's  performance  as  reflected  in  standardized  measures,  including, 
when  possible:  local,  State,  or  national  information  on  performance  of  comparable 
organizations. 

•  Data  from  other  outside  organizations,  such  as  Medicare  or  Medicaid  fee-for-service  data, 
data  from  other  health  plans,  and  local  or  national  public  health  reports  on  conditions  or 
risks  for  specified  populations. 

•  Data  from  surveys,  grievance  and  appeals  processes,  and  disenrollments  and  requests  to 
change  providers. 

Methods  of  Evaluation 
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Review  the  MCO's/PHP's  project  documentation  and,  as  needed,  the  above  data  sources  to 
assess  whether  the  MCO/PHP  had  selected  an  appropriate  study  topic.  In  general,  a  clinical  or 
non-clinical  issue  selected  for  study  should  affect  a  significant  portion  of  the  enrollees  (or  a 
specified  sub-portion  of  enrollees)  and  have  a  potentially  significant  impact  on  enrollee  health, 
functional  status  or  satisfaction.  The  topics  should  reflect  high-volume  or  high-risk  conditions  of 
the  population  served.  High-risk  conditions  may  occur  for  infrequent  conditions  or  services,  such 
as  when  a  pattern  of  unexpected  adverse  outcomes  are  identified  through  data  analysis.  High  risk 
also  exists  for  populations  with  special  health  care  needs,  such  as  children  in  foster  care,  adults 
with  disabilities  and  the  homeless.  Although  these  individuals  may  be  small  in  number,  their 
special  health  care  needs  places  them  at  high  risk. 

Consider  the  answers  to  the  following  questions  to  ascertain  if  the  MCO's/PHP's  PIP  reflected 
an  appropriate  study  topic. 

1 .  Was  the  topic  selected  by  the  MCO/PHP  identified  through  data  collection  and  analysis 
of  comprehensive  aspects  of  enrollee  needs,  care  and  services? 

Review  documentation  supplied  by  the  MCO/PHP  explaining  how  the  study  topic  was 
chosen.  Determine  the  extent  to  which  the  MCO/PHP  considered  enrollee  demographic 
characteristic  and  health  risks,  and  the  prevalence  of  the  chosen  topic  among,  or  the  need 
for  a  specific  service  by,  enrollees.  Determine  the  extent  to  which  the  explanation  is 
consistent  with  demographic  and  epidemiologic  information  on  the  MCO's/PHP's 
enrollees  or  consistent  with  information  on  similar  groups  in  the  MCO's/PHP's 
geographic  service  area. 

A  project  topic  also  may  be  suggested  by  patterns  of  inappropriate  utilization.  However, 
the  project  must  have  been  clearly  focused  on  identifying  and  correcting  deficiencies  in 
care  or  services  that  might  have  led  to  this  pattern,  such  as  inadequate  access  to  primary 
care,  rather  than  on  utilization  or  cost  issues  alone.  The  goal  of  the  project  should  be  to 
improve  health  outcomes.  Therefore,  it  is  acceptable  for  a  project  to  focus  on  patterns  of 
over  utilization  that  present  a  clear  threat  to  health  or  functional  status. 

Topics  to  be  studied  may  also  have  been  selected  in  the  basis  of  Medicaid  enrollee  input. 
To  the  extent  feasible,  MCOs  and  PHPs  are  encouraged  to  obtain  input  from  enrollees 
who  are  users  of  or  concerned  with  specific  focus  areas;  e.g.  priorities  in  the  area  of 
mental  health  or  substance  abuse  services  should  be  developed  in  consultation  with  users 
of  these  services  or  their  families. 

2.  Did  the  MCO's/PHP's  PIPs,  over  time,  address  a  broad  spectrum  of  key  aspects  of 
enrollee  care  and  services?  Did  the  MCO/PHP  select  topics  in  clinical  and  non-clinical 
focus  areas? 
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It  is  important  that,  over  time,  the  chosen  topic  areas  are  not  limited  to  recurring,  easily 
measured  subsets  of  health  care  needs  of  the  enrolled  population;  e.g.,  primary  preventive 
care  of  adults,  high  cost  care  of  adults.  Over  time,  a  MCO's/PHP's  PIPs  must  be 
representative  of  the  entire  spectrum  of  clinical  and  nonclinical  areas  associated  with  the 
MCO/PIP.  To  address  this,  HCFA's  NPRJVI  states  that  States  are  to  establish  MCO/PHP 
obligations  for  the  number  and  distribution  of  projects  among  required  clinical  and 
nonclinical  areas.  Clinical  focus  areas  include  prevention  and  care  of  acute  and  chronic 
conditions,  high-  volume  services,  and  high-risk  services.  Focusing  on  a  high-volume 
service,  as  opposed  to  a  clinical  condition,  and  how  it  can  be  improved  may  have  been 
the  target  of  an  MCO's/PHP's  performance  improvement  project.  The  improvement 
project  may  have  targeted  such  services  as  labor  and  delivery,  a  frequently  performed 
surgical  procedure,  or  different  surgical  or  invasive  procedures.  The  MCO/PHP  may  also 
target  high-risk  procedures  even  if  they  may  be  low  in  frequency;  e.g.,  care  received  from 
specialized  centers  inside  or  outside  of  the  organization's  network;  e.g.,  burn  centers, 
transplant  centers,  cardiac  surgery  centers.  It  could  also  assess  and  improve  the  way  in 
which  it  detects  which  of  its  members  have  special  health  care  needs  and  assess  these 
members'  satisfaction  with  the  care  received  from  the  organization. 

Finally,  clinical  PIPs  can  address  coordination  and  continuity  of  care.  Continuity  and 
coordination  of  care  both  address  the  manner  in  which  care  is  provided  when  a  patient 
receives  care  from  multiple  providers  and  across  multiple  episodes  of  care.  Such  studies 
may  be  disease  or  condition-specific  or  may  target  continuity  and  coordination  across 
multiple  conditions. 

Projects  in  non-clinical  areas  should  also  address,  over  time,  appeals,  grievance  and 
complaints;  as  well  as  access  to  and  availability  of  services.  Projects  in  non-clinical  areas 
can  focus  on  assessing  and  improving  the  accessibility  of  specific  services  or  services  for 
specific  conditions,  including  reducing  disparities  between  services  to  minorities  and 
service  to  other  members.  Projects  related  to  the  grievance  and  coverage  determination 
process  may  aim  either  to  improve  the  processes  themselves  or  to  address  an  underlying 
issues  in  care  or  services  identified  through  analysis  of  grievances  or  appeals. 

Step  2:  Review  the  study  question(s) 

Rationale 

It  is  important  for  the  MCO/PHP  to  clearly  state,  in  writing,  the  question(s)  the  study  is  designed 
to  answer.  Stating  the  question(s)  is  important  to  stay  focused  and  to  set  the  framework  for  data 
collection,  analysis  and  interpretation. 

Potential  Sources  of  Supporting  Information 

•         QI  study  documentation 
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•         Relevant  clinical  literature 


Method  of  evaluation 

Review  the  MCO's/PHP's  project  documentation  to  determine  whether  the  study  question(s)  was 
clearly  defined.  Consider  the  answers  to  the  following  question  to  determine  whether  a  clear 
study  question(s)  was  identified. 

1 .        Did  the  MCO/PHP  state  the  study  question(s)  clearly  in  writing? 

The  problem  to  be  studied  must  be  stated  as  clear,  simple,  answerable  question(s).  An 
example  of  a  vague  study  question  is: 

"Does  our  MCO/PHP  adequately  address  psychological  problems  in  patients 
recovering  from  myocardial  infarction?  " 

In  this  example,  it  is  not  clear  how  "adequately  address"  will  be  assessed. 
Furthermore,  "psychological  factors"  is  a  very  broad  term.  A  clearer  study 
question  could  be: 

"Does  doing  'x'  reduce  the  proportion  of  patients  with  myocardial  infarction  who 
develop  severe  emotional  depression  during  hospitalization?  " 

Step  3:  Review  the  selected  study  indicator(s) 

Rationale 

A  study  indicator  is  a  variable  reflecting  a  discrete  event  (e.g.,  an  older  adult  has/has  not  received 
a  flu  shot  in  the  last  12  months),  or  a  status  (e.g.,  an  enrollee's  hypertension  is/is  not  under 
control).  Each  project  should  have  one  or  more  quality  indicators  for  use  in  tracking 
performance  and  improvement  over  time.  All  indicators  must  be  objective,  clearly  and 
unambiguously  defined,  and  based  on  current  clinical  knowledge  or  health  services  research.  In 
addition,  all  indicators  must  be  capable  of  objectively  measuring  either  enrollee  outcomes  such 
as  changes  in  health  or  functional  status,  erirollee  satisfaction,  or  valid  proxies  of  these  outcomes. 

Indicators  can  be  few  and  simple,  many  and  complex,  or  any  combination  thereof,  depending  on 
the  study  question(s),  the  complexity  of  existing  practice  guidelines  for  a  clinical  condition,  and 
the  availability  of  data  and  availability  of  resources  to  gather  data. 

Indicator  criteria  are  the  set  of  rules  by  which  the  data  collector  or  reviewer  determines  whether 
an  indicator  has  been  met.  Pilot  or  field  testing  is  helpful  to  the  development  of  effective 
indicator  criteria.  Such  testing  allows  the  opportunity  to  add  criteria  that  might  not  have  been 
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anticipated  in  the  design  phase.  In  addition,  criteria  are  often  refined  over  time,  based  on  results 
of  previous  studies.  However,  if  criteria  are  changed  significantly,  the  method  for  calculating  an 
indicator  will  not  be  consistent  and  performance  on  indicators  will  not  be  comparable  over  time. 
It  is  important,  therefore,  for  the  indicator  criteria  to  be  developed  as  fully  as  possible  during  the 
design  and  field  testing  of  data  collection  instruments. 

Potential  Sources  of  Supporting  Information 

•  Clinical  and  non-clinical  practice  guidelines 

•  Administrative  data 

•  Medical  records 

Methods  of  Evaluation 

Review  the  MCO's/PHP's  project  documentation  to  assess  whether  appropriate  study  indicators 
were  used.  Use  the  following  questions  to  help  assess  study  indicators. 

1 .       Did  the  study  use  objective,  clearly  and  unambiguously  defined,  measurable  indicators? 

When  indicators  exist  that  are  generally  used  within  the  public  health  community  or  the 
managed  care  industry  (such  as  NCQA's  Health  Plan  Employer  Data  and  Information  Set 
(HEDIS)  or  the  Foundation  for  Accountability's  (FACCT)  measures)  and  these  indicators 
are  applicable  to  the  topic,  use  of  those  indicators  is  preferred.  However,  indicators  may 
be  developed  by  the  MCO/PHP  on  the  basis  of  current  clinical  knowledge  or  health 
services  research.  When  a  MCO/PHP  develops  its  own  indicators,  it  must  be  able  to 
document  the  basis  on  which  it  adopted  an  indicator. 

Consider  the  following  list  of  key  characteristics  to  determine  if  meaningful  indicators 
were  developed. 

►  Was/were  the  indicator(s)  related  to  identified  health  care  guidelines 
pertinent  to  the  study  question? 

►  Was  this  an  important  aspect  of  care  to  monitor  that  made  a  difference  to 
the  MCO's/PHP's  beneficiaries? 

►  Were  the  data  available  either  through  administrative  data,  medical  records 
or  other  readily  accessible  sources? 

►  Did  limitations  on  the  ability  to  collect  the  data  skew  the  results? 

►  Did  these  indicators  require  explicit  or  implicit  criteria?  The  MCO/PHP 
must  consider  the  specificity  of  the  criteria  used  to  determine  compliance 
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with  an  indicator  or  with  a  performance  measure  related  to  the  indicator. 
The  greater  number  of  people  involved  in  data  collection  and  analysis,  the 
greater  the  need  for  more  explicit,  or  precise,  data  collection  and  indicator 
criteria  to  obtain  inter-reviewer  reliability.  The  more  specific  the  criteria, 
the  easier  the  data  collection  process  will  be,  because  staff  will  not  need 
extensive  training.  An  example  of  an  explicit  criterion  for  an 
immunization  study,  is: 

•  Documentation  of  refusal  by  parent  to  have  a  child  immunized 
through  nurses  notes  and/or  signed  refusal  by  the  parent  in  the 
medical  record. 


Implicit  criteria  may  require  a  high  degree  of  professional  clinical 
judgement,  and  therefore,  may  be  time-consuming  and  expensive.  An 
example  of  an  implicit  criterion  for  an  immunization  study,  is: 

•         Medical  contraindications  for  receiving  childhood  immunizations. 

Specific  indicators  do  not  always  need  to  be  set  at  the  outset  of  a  performance 
improvement  project.  There  may  be  instances  in  which  a  project  would  begin  with  more 
general  collection  and  analysis  of  baseline  data  on  a  topic,  and  then  narrow  its  focus  to 
more  specific  indicators  for  measurement,  intervention  and  reevaluation.  The  success  of 
the  project  is  assessed  in  terms  of  the  indicators  ultimately  selected. 

2.        Did  the  MCO's/PHP's  indicators  measure  changes  in  health  status,  functional  status,  or 
enrollee  satisfaction,  or  valid  proxies  of  these  outcomes? 

The  objective  of  a  performance  improvement  project  should  be  to  improve  outcomes, 
defined  as  objective  measures  of  patient  health,  functional  status  or  satisfaction  following 
the  receipt  of  care  or  services.  Indicators  selected  for  a  topic  in  a  clinical  focus  area 
should  include  at  least  some  measure  of  change  in  health  status  or  functional  status  or 
process  of  care  proxies  for  these  outcomes.  Indicators  may  also  include  measures  of 
satisfaction. 

It  is  recognized,  however,  that  relatively  few  standardized  performance  measures  actually 
address  outcomes.  Even  when  outcome  measures  are  available,  their  utility  as  quality 
indicators  may  be  limited  because  outcomes  can  be  significantly  influenced  by  factors 
outside  of  the  organization's  control,  such  as  poverty,  genetics,  environment.  Quality 
indicators  do  not  always  need  to  be  outcome  measures.  Process  measures  are  acceptable 
as  long  as  it  can  be  shown  that  there  is  strong  clinical  evidence  that  the  process  being 
measured  is  meaningfully  associated  with  outcomes.  To  the  extent  possible,  this 
determination  should  be  based  on  published  guidelines  that  support  the  association  and 
that  cite  evidence  from  randomized  clinical  trials,  case  control  studies,  or  cohort  studies. 
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Although  published  evidence  is  generally  required,  there  may  be  certain  areas  of  practice 
for  which  empirical  evidence  of  process/outcome  linkage  is  limited.  At  a  minimum,  it 
should  be  demonstrated  that  there  is  a  consensus  among  relevant  practitioners  with 
expertise  in  the  defined  area  as  to  the  importance  of  a  given  process. 

While  it  is  recommended  to  consider  enrollee  satisfaction  as  an  important  outcome  of 
care  in  clinical  areas,  improvement  in  satisfaction  should  not  be  the  sole  demonstrable 
outcome  of  a  project  in  any  of  these  areas.  Some  improvement  in  health  or  functional 
status  should  also  be  measured.  For  projects  in  non-clinical  areas,  use  of  health  or 
functional  status  indicators  also  is  generally  preferred,  particularly  for  projects  addressing 
access  to  and  availability  of  health  care  services.  However,  there  may  be  some  non- 
clinical projects  for  which  enrollee  satisfaction  indicators  alone  are  sufficient. 

Step  4:  Review  the  identified  study  population 

Rationale 

Once  a  topic  has  been  selected,  measurement  and  improvement  efforts  must  be  system-wide;  i.e., 
each  project  must  represent  the  entire  Medicaid  enrolled  population  to  which  the  PIP  study 
indicators  apply.  Once  that  population  is  identified,  the  MCO/PHP  must  decide  whether  to 
review  data  for  that  entire  population  or  use  a  sample  of  that  population.  Sampling  is  acceptable 
as  long  as  the  samples  are  representative  of  the  identified  population  (see  Activity  5). 

Potential  Sources  of  Supporting  Information 

•  Data  on  the  Medicaid  enrolled  population  that  enumerates  the  numbers  of  enrollees  to 
which  the  study  topic  and  indicators  apply.  This  would  include  demographic  information 
from  MCO/PHP  enrollment  files  and  MCO/PHP  utilization,  diagnostic  and  outcome 
information,  such  as  services,  procedures,  admitting  and  encounter  diagnoses,  adverse 
incidents  (such  as  deaths,  avoidable  admissions,  or  readmissions),  and  patterns  of 
referrals  or  authorization  requests. 

•  Other  data  bases,  as  needed;  e.g.,  pharmacy  claims  data  to  identify  patients  taking  a 
defined  number  of  a  specific  medication(s)  during  a  specific  enrollment  period. 

Methods  of  Evaluation 

Review  the  study  description  and  methodology  to  assess  whether  the  study  clearly  identified  the 
study  population.  Consider  the  answers  to  following  questions  to  assess  the  extent  to  which  the 
MCO/PHP  clearly  identified  the  study  population. 

1 .       How  did  the  MCO/PHP  define  the  study's  "at  risk"  population? 
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•  Did  the  MCO/PHP  clearly  define  all  individuals  to  whom  the  identified  study 
question(s)  and  indicators  are  relevant? 

•  Did  the  MCO/PHP  include  the  entire  study  population  or  use  a  sample  in  the 
study?  The  organization's  decision  may  have  been  determined  by  the  resources 
available  to  analyze  the  data.  If  the  organization  is  capable  of  collecting  and 
analyzing  data  through  an  automated  data  system,  it  might  be  possible  to  study  the 
whole  population  because  many  of  the  data  collection  and  analysis  steps  can  be 
automated.  If  the  data  must  be  collected  manually,  sampling  may  be  more 
realistic. 

•  Did  the  definition  of  the  study  population  include  any  requirements  for  the  length 
of  the  study  populations'  members  enrollment  in  the  MCO  or  PHP? 

•  Was  the  study  population  enrolled  for  a  sufficient  length  of  time  to  demonstrate 
the  MCO/PHP  delivery  system  at  work?  The  required  length  of  time  will  vary 
depending  on  the  study  topic  and  study  indicators. 

•  If  the  MCO/PHP  studied  the  entire  population,  did  its  data  collection  approach 
truly  capture  all  enrollees  to  which  the  study  question  applied? 

If  the  MCO/PHP  used  a  sample,  go  to  Step  5.  If  the  MCO/PHP  studied  the  entire  population, 
skip  Step  5  and  go  to  Step  6. 

Step  5:  Review  Sampling  Methods 

Rationale 

If  the  MCO/PHP  used  a  sample  to  select  members  of  the  study,  proper  sampling  techniques  need 
to  be  used  to  provide  valid  and  reliable  (and  therefore  generalizable)  information  on  the  quality 
of  care  provided.  When  conducting  a  study  designed  to  estimate  the  rates  at  which  certain  events 
occur,  the  sample  size  has  a  large  impact  on  the  level  of  statistical  confidence  in  the  study 
estimates.  Statistical  confidence  is  a  numerical  statement  of  the  probable  degree  of  certainty  or 
accuracy  of  an  estimate.  In  some  situations,  it  expresses  the  probability  that  a  difference  could  be 
due  to  chance  alone.  In  other  applications,  it  expresses  the  probability  of  the  accuracy  of  the 
estimate.  For  example,  a  study  may  report  that  a  disease  is  estimated  to  be  present  in  35%  of  the 
population.  This  estimate  might  have  a  95%  level  of  confidence,  plus  or  minus  five  percentage 
points.  This  means  that  we  are  95%  sure  that  between  30-40  percent  of  the  population  has  the 
disease. 

The  true  prevalence  or  incidence  rate  for  the  event  in  the  population  may  not  be  known  for  the 
first  time  a  topic  is  studied.  In  such  situations,  the  most  prudent  course  of  action  is  to  assume 
that  a  maximum  sample  size  is  needed  to  establish  a  statically  valid  baseline  for  the  plan. 
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Potential  Sources  of  Supporting  Information 


•  Data  on  enrollee  characteristics  relevant  to  health  risks  or  utilization  of  clinical  and  non- 
clinical services,  including  age,  sex,  race/ethnicity/language  and  functional  status. 

•  Utilization,  diagnostic  and  outcome  information,  such  as  services,  procedures,  admitting 
and  encounter  diagnoses,  adverse  incidents  (such  as  deaths,  avoidable  admissions,  or 
readmissions),  and  patterns  of  referrals  or  authorization  requests. 

•  Other  information  as  needed,  such  as  pharmacy  claims  data  to  identify  patients  taking  a 
defined  number  of  a  specific  medication(s)  during  a  specific  enrollment  period. 

Methods  of  Evaluations 

Review  the  study  description  and  methodology  to  ensure  that  proper  sampling  techniques  were 
applied.  Consider  the  answers  to  the  following  questions  in  evaluating  the  soundness  of  the 
MCO's/PHP's  approach  to  sampling. 

1 .  Did  the  methods  used  by  the  MCO/PHP  to  calculate  the  needed  sample  size  consider  and 
specify  the  true  (or  estimated)  frequency  of  occurrence  of  the  event,  the  confidence 
interval  to  be  used,  and  the  acceptable  margin  of  error? 

2.  Were  the  selected  confidence  interval  and  margin  of  error  reasonable? 

3.  Did  the  MCO/PHP  employ  valid  sampling  techniques? 

•         There  are  two  basic  categories  of  statistical  sampling  methods  —  probability 
sampling  and  nonprobability  sampling. 

Probability  (or  random)  sampling  methods  leave  selection  of  population  units 
totally  to  chance,  and  not  to  preference  on  the  part  of  the  individuals  conducting 
or  otherwise  participating  in  the  study.  Biases  are  removed  in  these  methods. 
There  are  several  types  of  probability  (or  random)  sampling  that  can  be  used  by 
the  MCO/PHP: 

►  In  simple  random  sampling,  all  members  of  the  study  population  have  an 
equal  chance  of  being  selected  for  the  sample.  Population  members  are 
generally  numbered,  and  random  numbers  generated  by  computer  are  used 
to  select  units  from  the  population. 

►  Systematic  random  sampling  -  the  basic  principle  is  to  select  every  Nth 
unit  in  a  list.  This  can  be  used  when  a  sampling  frame  is  organized  in  a 


HCFA/CMSO/QPMG/DQSM  -  September  11,  2000  DRAFT 


Page  15 


way  that  does  not  bias  the  sample.  Steps  in  conducting  a  systematic 
sampling  are: 

1)  Construct  a  comprehensive  sampling  frame  (e.g.,  list  of  all 
beneficiaries). 

2)  Divide  the  size  of  the  sampling  frame  by  the  required  sample  size 
to  produce  a  sampling  interval  or  skip  interval  (e.g.,  if  there  are 
250  beneficiaries  and  a  sample  of  25  is  needed,  then  divide  250/25 
=  10). 

3)  From  a  random  number  table  select  a  random  number  between  1 
and  10. 

4)  Count  down  the  list  to  get  the  Nth  name  (i.e.,  the  #  identified  in 
step  3). 

5)  Skip  down  10  names  on  the  list  and  select  a  second  name.  Repeat 
the  process  as  many  times  as  needed  until  the  required  sample  size 
has  been  reached. 

►  Stratified  random  sampling  is  used  when  the  target  population  consists  of 
non-overlapping  sub-groups  or  strata.  Typically  this  is  used  if  the 
population  is  homogeneous  (same)  within  a  strata  and  heterogeneous 
(different)  between  a  strata.  Stratified  random  sampling  requires  more 
information  about  the  population  and  also  requires  a  larger  overall  sample 
size  than  simple  random  sampling. 

►  Cluster  sampling  is  used  when  a  comprehensive  sampling  frame  is  NOT 
available.  Units  in  the  population  are  gathered  or  classified  into  groups, 
similar  to  stratified  sampling.  Unlike  the  stratified  sampling  method,  the 
groups  must  be  heterogeneous  within  themselves  with  respect  to  the 
characteristic  being  measured.  This  method  requires  prior  knowledge 
about  the  population.  Once  clusters  are  identified,  a  random  sample  of 
clusters  are  selected. 

Non-probability  sampling  methods  are  based  on  personal  choice,  rather  than 
chance;  therefore  some  bias  can  be  expected.  There  are  several  types  of  non- 
probability  sampling  that  can  be  used: 

►  Judgement  sampling  involves  constructing  a  sample  based  on  including 
units  in  the  sample  if  they  are  thought  (or  judged)  to  be  representative  of 
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the  population.  By  doing  so,  the  sample  is  constructed  to  be  a  mini- 
population. 

►  Convenience  sampling  uses  units  that  are  readily  or  conveniently 
available.  For  example,  if  the  objective  were  beneficiary  opinions 
regarding  a  group  practice,  patients  in  the  office  on  any  given  day  or 
during  a  specific  month  could  be  interviewed. 

►  Quota  sampling  ensures  that  units  in  the  sample  appear  in  the  same 
proportion  as  in  the  population.  For  instance,  if  a  certain  target  population 
consisted  of  55%  female  and  45%  male,  the  quota  sample  would  require  a 
similar  female/male  distribution. 

Step  6:  Review  the  MCO's/PHP's  data  collection  procedures 

Rationale 

Procedures  used  by  the  MCO/PHP  to  collect  data  for  its  PIP  must  ensure  that  the  data  collected  on 
the  PIP  indicators  are  valid  and  reliable.  Validity  is  an  indication  of  the  accuracy  of  the 
information  obtained  from  a  study.  Reliability  is  an  indication  of  the  repeatability  or 
reproducibility  of  a  measurement.  The  MCO  or  PHP  should  have  employed  a  data  collection  plan 
that  included: 

•  clear  identification  of  the  data  to  be  collected, 

•  identification  of  the  data  sources  and  how  and  when  the  baseline  and  repeat  indicator  data 
will  be  collected, 

•  specification  of  who  will  collect  the  data,  and 

•  identification  of  instruments  used  to  collect  the  data. 

When  data  is  collected  from  automated  data  systems,  development  of  specifications  for  automated 
retrieval  of  the  data  is  necessary.  When  data  are  obtained  from  visual  inspection  of  medical 
records  or  other  primary  source  documents,  several  steps  should  be  taken  to  ensure  the  data  are 
consistently  extracted  and  recorded: 

1 .       The  key  to  successful  manual  data  collection  is  in  the  selection  of  the  data  collection  staff. 
Appropriately  qualified  personnel,  with  conceptual  and  organizational  skills,  must  be  used 
to  abstract  the  data;  however,  the  specific  skills  will  vary  with  the  nature  of  the  data  being 
collected  and  the  degree  of  professional  judgment  required.  For  example,  when  data 
collection  involves  searching  throughout  the  medical  record  to  find  and  abstract 
information  or  the  abstractor  must  judge  whether  clinical  criteria  were  met,  experienced 
clinical  staff,  such  as  registered  nurses  should  collect  the  data.  However,  when  the 
abstraction  involves  verifying  the  presence  of  a  diagnostic  test  report,  trained  medical 
assistants  or  medical  records  clerks  may  be  used. 


HCFA/CMSO/QPMG/DQSM  -  September  11,  2000  DRAFT 


Page  17 


2.  Clear  guidelines  for  obtaining  and  recording  data  must  be  established,  especially  if 
multiple  reviewers  are  used  to  perform  this  activity.  The  MCO/PHP  should  have 
determined  the  necessary  qualifications  of  the  data  collection  staff  before  finalizing  the 
data  collection  instrument.  The  abstractor  will  need  fewer  clinical  skills  if  the  data 
elements  within  the  data  source  are  more  clearly  defined.  Defining  a  glossary  of  terms  for 
each  project  should  be  a  part  of  the  training  of  abstractors  to  ensure  consistent 
interpretation  among  and  between  the  project  staff. 

3.  The  number  of  data  collection  staff  to  be  used  for  a  given  project  affects  the  reliability  of 
the  data.  A  smaller  number  of  staff  promotes  inter-reviewer  reliability;  however,  it  may 
also  increase  the  amount  of  time  it  takes  to  complete  this  task.  Intra-rater  reliability  (i.e., 
reproducibility  of  judgements  by  the  same  abstractor  at  a  different  time)  should  also  be 
considered. 

Potential  Sources  of  Supporting  Information 

•  List  of  sources  of  data  used  in  the  study 

•  If  medical  record  review,  or  other  manual  data  collection  was  used  to  produce  study  data: 

-  data  recording  forms 

-  instructions  to  data  collectors 

•  If  automated  data  collection  was  used,  an  algorithm  showing  the  steps  in  the  production  of 
quality  indicators  and  other  relevant  data  collection 

Methods  of  Evaluation 

Evaluation  of  the  MCO's/PHP's  data  collection  procedures  can  be  determined  through  two 
processes:  assessing  the  study's  approach  to  data  collection  (discussed  in  this  step)  and  conducting 
a  verification  sample  of  the  study's  findings  (discussed  in  ACTIVITY  II).  Consider  the  answers  to 
the  following  questions  in  determining  the  soundness  of  data  collection  procedures. 

1 .  Did  the  study  design  clearly  specify  the  data  to  be  collected? 

Accurate  measurement  depends  on  clearly  defined  data  elements.  Data  elements  must  be 
carefully  specified  with  unambiguous  definitions.  When  descriptive  terms  are  used  (e.g., 
high,  low,  normal),  numerical  definitions  are  established  for  each  term.  The  units  of 
measure  must  also  be  specified  (e.g.,  pounds,  kilograms,  etc.). 

2.  Did  the  study  design  clearly  specify  the  sources  of  data? 

Data  sources  vary  considerably  and  depend  upon  the  selected  topic  and  indicators. 
Similarly,  the  topic  and  indicators  will  reflect  not  just  the  clinical  and  research 
considerations,  but  also  the  available  MCO/PHP  data  sources.  Sources  can  include: 
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beneficiary  medical  records,  tracking  logs,  encounter  and  claims  systems,  provider 
interviews,  beneficiary  interviews  and  surveys. 

3.        Did  the  study  design  specify  a  systematic  method  of  collecting  valid  and  reliable  data  that 
represents  the  entire  population  to  which  the  study's  indicators  apply? 

The  MCO's/PHP's  PIP  study  may  use  automated  or  manual  data  collection  methods 
depending  on  the  resources  available.  If  an  automated  data  collection  system  was  utilized, 
the  degree  of  completeness  of  the  data  in  the  automated  system  is  always  a  concern.  For 
example,  for: 

►  Inpatient  data:  Did  the  data  system  capture  all  inpatient  admissions? 

►  Primary  care  data:  Did  primary  care  providers  submit  encounter  data  for  all 
encounters? 

►  Specialty  care  data:  Did  specialty  care  providers  submit  encounter  data  for 
all  encounters? 

►  Ancillary  services  data:  Did  ancillary  service  providers  submit  encounter  or 
utilization  data  for  all  services  provided? 

The  study's  design  and  methodology  should  include  an  estimation  of  the  degree  of 
completeness  of  the  automated  data  used  for  the  PIP  study  indicators. 

Manual  data  collection  may  be  the  only  feasible  option  for  many  MCOs/PHPs  and  for 
many  topics  selected.  The  beneficiary  medical  record  is  the  most  frequently  used  data 
source.  Other  manual  systems  which  might  contain  sources  of  information  may  include 
clinical  tracking  logs,  registries,  complaint  logs,  and  manual  claims.  When  evaluating 
manual  data  collection,  the  following  issues  should  be  considered: 

►  Did  the  MCO/PHP  use  qualified  staff  and  personnel  to  collect  the  data? 

Clinical  knowledge  and  skills  were  addressed,  including  good  conceptual 
skills,  organization  skills,  thoroughness,  and  strong  documentation  skills. 

►  Did  the  MCO/PHP  use  instruments  for  data  collection  that  provide  for 
reliable  and  accurate  data  collection  over  the  time  periods  studied? 

If  manual  data  collection  was  performed,  the  data  collection  instrument(s) 
should  be  clear  and  promote  inter-rater  reliability.  An  important  part  of 
designing  the  data  collection  form  is  developing  instructions  or  guidelines 
for  data  collection  staff.  Form  design  is  particularly  important  when  staff 
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not  involved  in  the  study  design  perform  data  collection.  Instructions 
should  be  clearly  and  succinctly  written  and  should  provide  an  overview  of 
the  study,  specific  instructions  on  how  to  complete  each  section  of  the  form 
and  general  guidance  on  how  to  handle  situations  not  covered  by  the 
instructions. 

4.        Did  the  study  design  prospectively  specify  a  data  analysis  plan  which  reflected  the 
following  considerations? 

•  Whether  qualitative  or  quantitative  data,  or  both,  were  to  be  collected. 

Qualitative  data  describes  characteristics  or  attributes  by  which  persons  or  things 
can  be  classified;  for  example,  sex,  race,  poverty  level,  or  the  presence  or  absence 
of  a  specific  disease.  Calculation  of  proportions  and  calculation  of  rates  are  the 
two  most  common  qualitative  measures. 

Quantitative  data  are  concerned  with  numerical  variables  such  as  height,  weight 
and  blood  levels.  The  methods  by  which  the  data  are  analyzed  and  presented  will 
vary  by  type  of  data.  Quantitative  data  require,  at  a  minimum,  simple  descriptive 
statistics  such  as  measures  of  central  tendency  (i.e.,  mean,  median  or  mode)  and 
measure  of  variability  (i.e.,  range  or  standard  deviation). 

•  Whether  the  data  were  to  be  collected  on  the  entire  population  or  a  sample. 

•  Whether  the  measurements  obtained  from  the  data  collection  activity  were  to  be 
compared  to  the  results  of  previous  or  similar  studies.  If  so,  the  data  analysis  plan 
should  have  considered  evaluating  the  comparability  of  the  studies  and  identified 
the  appropriate  statistical  tests  to  be  used  to  compare  studies. 

•  Whether  the  performance  improvement  project  was  to  be  compared  to  the 
performance  of  an  individual  health  plan,  number  of  health  plans,  or  different 
provider  sites.  Comparing  the  performance  of  multiple  entities  involves  greater 
statistical  design  and  analytical  considerations  than  those  required  for  a  study  of  a 
single  entity,  such  as  a  health  plan. 

Step  7:  Assess  the  MCO's/PHP's  Improvement  Strategies. 

Rationale 

Real,  sustained  improvements  in  care  result  from  a  continuous  cycle  of  measuring  performance, 
analyzing  performance,  and  developing  and  implementing  system-wide  improvements  in  care. 
Actual  improvements  in  care  depend  far  more  on  thorough  analysis  and  implementation  of 
appropriate  solutions  than  on  any  other  steps  in  the  process. 
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An  improvement  strategy  is  defined  as  an  intervention  designed  to  change  behavior  at  an 
institutional,  practitioner  or  beneficiary  level.  The  effectiveness  of  the  intervention  activity  or 
activities  can  be  determined  by  measuring  the  MCO's/PHP's  change  in  performance,  according  to 
predefined  quality  indicators.  Interventions  are  key  to  an  improvement  project's  ability  to  bring 
about  improved  health  care.  Appropriate  interventions  must  be  identified  and/or  developed  for 
each  performance  improvement  project,  to  assure  the  likelihood  of  effecting  measurable  change. 

If  repeat  measures  of  quality  improvement  indicate  that  QI  actions  were  not  successful,  i.e.,  did 
not  achieve  significant  improvement,  the  problem-solving  process  begins  again  with  data  analysis 
to  identify  possible  causes,  propose  and  implement  solutions,  and  so  forth.  If  QI  actions  were 
successful,  the  new  processes  should  be  standardized  and  monitored. 

Potential  Sources  of  Supporting  Information 

•  Current  project  baseline  data 

•  Previous  project  data  (if  available) 

•  Results  of  clinical  and  literature  research 

•  Project  evaluation  results  completed  by  evaluators 

Methods  of  Evaluation: 

Consider  the  answers  to  the  following  question  to  help  determine  the  extent  to  which  appropriate 
interventions  were  addressed. 

1 .       Did  the  MCOs/PHPs  undertake  interventions  related  to  causes/barriers  identified  through 
data  analysis  and  QI  processes? 

It  is  expected  that  interventions  associated  with  improvement  on  quality  indicators  will  be 
system  interventions,  i.e.,  educational  efforts,  changes  in  policies,  targeting  of  additional 
resources,  or  other  organization-wide  initiatives  to  improve  performance.  Interventions 
that  might  have  some  short  term  effect,  but  that  are  unlikely  to  induce  permanent  change 
(such  as  a  one-time  reminder  letter  to  physicians  or  beneficiaries)  are  insufficient. 

An  MCO/PHP  is  not  required  to  demonstrate  conclusively  (for  example,  through 
controlled  studies)  that  a  change  in  an  indicator  is  the  effect  of  its  intervention;  it  is 
sufficient  to  show  that  an  intervention  occurred  that  might  reasonably  be  expected  to  affect 
the  results.  Nor  is  the  MCO/PHP  required  to  undertake  data  analysis  to  correct  for  secular 
trends  (changes  that  reflect  continuing  growth  or  decline  in  a  measure  as  a  result  of 
external  forces  over  an  extended  period  of  time).  To  the  extent  feasible,  however,  the 
MCO/PHP  should  be  able  to  demonstrate  that  its  data  have  been  corrected  for  any  major 
confounding  variables  with  an  obvious  impact  on  the  outcomes.  The  MCO's/PHP's 
interventions  are  determined  to  reasonably  result  in  measured  improvement. 
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Step  8:  Review  MCO/PHP  data  analysis  and  interpretation  of  study  results. 


Rationale 


Review  of  MCO/PHP  data  analysis  begins  with  examining  the  MCO's/PHP's  calculated  plan 
performance  on  clinical  or  non-clinical  indicators.  The  review  examines  the  appropriateness  of, 
and  the  MCO's/PHP's  adherence  to,  the  statistical  analysis  techniques  defined  in  the  data  analysis 
plan. 

Potential  Sources  of  Supporting  Information 

•  Baseline  project  indicator  measurements 

•  Repeat  project  indicator  measurements 

•  Industry  benchmarks 

•  Analytic  reports  of  PIP  results  by  the  MCO/PHP 
Methods  of  Evaluation 

Consider  the  answers  to  each  of  the  following  to  assess  the  extent  to  which  MCO/PHP  PIP  data 
analysis  and  interpretation  was  appropriate  and  valid. 

1 .  Did  the  MCO/PHP  conduct  an  analysis  of  the  findings  according  to  its  data  analysis  plan? 

2.  Did  the  MCO/PHP  present  numerical  PIP  results  and  findings  data  in  a  way  that  provides 
accurate,  clear  and  easily  understood  information? 

3.  Following  the  data  analysis  plan,  did  the  analysis  identify: 


► 


initial  and  repeat  measurements  of  the  prospectively  identified  clinical 
indicators  for  the  project? 


► 


the  statistical  significance  of  any  differences  between  the  initial  and  repeat 
measurements? 


► 


factors  that  influence  the  comparability  of  initial  and  repeat  measurements? 


factors  that  threaten  the  internal  or  external  validity  of  the  findings? 


4. 


Did  the  MCO's/PHP's  analysis  of  the  study  data  include  an  interpretation  of  the  extent  to 
which  its  PIP  was  successful  and  what  follow-up  activities  were  planned  as  a  result? 
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Interpretation  and  analysis  of  the  study  data  should  be  based  on  continuous  improvement 
philosophies  and  reflect  an  understanding  that  most  problems  result  from  failures  of 
administrative  or  delivery  system  processes,  not  failures  of  individuals  within  the  system. 
Interpreting  the  data  should  involve  developing  a  hypotheses  about  the  causes  of  less-than- 
optimal  performance  and  collecting  data  to  validate  the  hypotheses. 

Step  9:  Assess  whether  improvement  reported  as  part  of  a  PIP  is  significant. 

Rationale 

HCFA's  NPRM  proposed  to  require  that  an  MCO's/PHP's  PIPs  involve  interventions  that  achieve 
improvement  that  is  "significant."  The  degree  to  which  an  intervention  is  statistically  "significant" 
can  be  measured  quantitatively  through  statistical  calculations.  While  this  protocol  does  not 
specify  a  level  of  statistical  significance  that  must  be  met,  it  does  require  that  EQROs  assess  the 
extent  to  which  any  changes  in  performance  reported  by  an  MCO/PHP  as  a  "significant"  can  be 
found  to  be  statistically  significant.  States  may  choose  to  establish  their  own  numerical  thresholds 
for  finding  reported  improvements  to  be  "significant." 

Potential  Sources  of  Supporting  Information 

•  Baseline  and  repeat  measures  on  quality  indicators 

•  Tests  of  statistical  significance  calculated  on  baseline  and  repeat  indicator  measurements 

•  Benchmarks  for  quality  specified  by  the  State  Medicaid  agency,  HCFA  for  Medicaid,  or 
found  in  industry  standards 

Methods  of  Evaluation 

At  the  point  in  time  an  MCO/PHP  is  far  enough  into  its  improvement  cycle,  review 
documentation  to  determine  the  extent  to  which  improvement  occurred.  The  following  elements 
should  be  considered  in  making  the  evaluation. 

1 .  Through  repeated  measurement  of  the  quality  indicators  selected  for  the  project, 
meaningful  change  in  performance  relative  to  the  performance  observed  during  baseline 
measurement  must  be  demonstrated.  The  repeat  measurement  should  use  the  same 
methodology  as  the  baseline  measurement,  except  that,  when  baseline  data  was  collected 
for  the  entire  population  at  risk,  the  repeat  may  instead  use  a  reliable  sample.  Performance 
using  the  identified  indicators  can  be  measured  by  collecting  information  on  all 
individuals,  encounters  or  episodes  of  care  to  which  the  indicator  is  applicable  (a  census) 
or  by  collecting  information  on  a  representative  subset  of  individuals,  encounters, 
providers  of  care,  etc. 

2.  A  project  which  measured  performance  on  quality  indicators  by  collecting  data  on  all  units 
of  analysis  in  the  population  to  be  studied  should  demonstrate  significant  improvement  by 
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achieving  a  level  of  improvement  defined  by  the  State  Medicaid  agency  or  defined  by  the 
MCO  or  PHP  as  approved  by  the  State  Medicaid  agency; 

3.        When  a  project  measured  performance  on  quality  indicators  by  collecting  data  on  a  subset 
(sample)  of  the  units  of  analysis  in  the  population  to  be  studied,  was  significant 
improvement  demonstrated  by  achieving  the  benchmarks  specified  previously,  or  was  the 
quantitative  improvement  demonstrated  in  the  repeated  measurements  statistically 
significant. 

Activity  10:  Assess  whether  the  MCO/PHP  has  sustained  its  documented  improvement 

Rationale: 

HCFA's  NPRM  proposed  to  require  that  the  MCO's  interventions  must  achieve  improvement  that 
is  significant  and  "sustained  over  time." 

Potential  Sources  of  Supporting  Information 

•  Baseline  and  first  repeated  measurements  on  quality  indicators 

•  Additional  measurements  on  quality  indicators  made  after  the  first  repeat  measurement 
Methods  of  Evaluation 

Review  of  the  remeasurement  documentation  is  required  to  assure  that  the  improvement  on  a 
project  is  sustained.  Consider  the  answer  to  the  following  question  in  making  a  determination  as 
to  whether  the  improvement  was  sustained. 

1 .        Was  the  MCO/PHP  able  to  demonstrate  sustained  improvement  through  repeated 
measurements  over  comparable  time  periods? 

The  MCO/PHP  should  repeat  measurements  of  the  indicators  after  the  initial  indicator 
measurement.  It  is  recognized  that  because  of  random  year-to-year  variation,  population 
changes,  and  sampling  error,  performance  on  any  given  individual  measure  may  decline  in 
the  second  measurement.  However,  when  all  of  the  MCOs/PHP's  repeat  measurements 
for  a  given  review  are  taken  together,  this  decline  should  not  be  statistically  significant  and 
should  never  be  statistically  significant  after  two  remeasurement  periods. 

ACTIVITY  2:        VERIFY  STUDY  FINDINGS  (OPTIONAL) 

Rationale 

In  addition  to  reviewing  the  methodology  and  findings  of  a  MCO's/PHP's  PIP,  an  additional 
activity  that  States  may  want  their  EQRO  to  address  for  some  or  all  of  the  PIPs  that  the  EQRO 
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validates  is  to  verify  the  actual  findings  of  the  PIP;  i.e.,  determine  if  the  initial  and  repeat 
measurements  of  the  quality  indicators  are  accurate.  While  this  activity  can  be  resource  intensive 
and  not  be  feasible  to  perform  for  every  PIP  that  an  EQRO  is  to  validate,  verification  activities 
can  provide  added  confidence  in  reported  MCO/PHP  PIP  results  because  they  provide  evidence 
that  a  given  PIP  is  free  from  bias. 

Potential  Data  Sources  Needed  for  Verification  Activities 

•  Current  project  data  and  findings 

•  Medical  records 

•  MCO/PHP  administrative  data 

•  Beneficiary  interviews  and  surveys 

•  An  assessment  of  the  MCO's/PHP's  Information  System  (IS) 
Methods  of  Evaluation 

The  key  focus  in  this  activity  is  validating  the  processes  through  which  data  needed  to  produce 
quality  indicators  was  obtained,  converted  to  information,  and  analyzed.  How  to  verify  quality 
indicator  findings  depends  on  whether  the  data  was  obtained  through  review  and  abstraction  of 
medical  records  or  produced  through  an  MCO's  /PHP's  automated  Information  Systems  (IS): 

Verification  of  data  obtained  through  medical  record  review:  Verification  of  quality 
indicators  produced  through  medical  record  review  can  be  achieved  by  conducting  a  re-abstraction 
of  a  small  subset  (validation  sample)  of  the  records  that  provided  the  data  for  the  quality 
indicators  used  in  the  study.  Data  retrieval  and  analysis  will  be  conducted  on  a  small  scale,  with 
the  validation  sample  following  the  same  abstracting  rules  of  the  original  study.  Statistical 
correlations  then  will  be  made  between  the  validation  sample  and  the  original  study  data. 

A  wide  variety  of  statistical  methods  can  be  applied  to  assess  the  degree  of  correlation  between 
the  study  and  validation  measures.  Two  recommended  methods  are  the  Pearson  correlation 
coefficient  for  continuous  data  (e.g.,  age,  income,  etc)  and  the  Kappa  statistic  for  categorical  data 
(e.g.,  gender,  race,  etc.). 

Verification  of  data  obtained  though  MCO/PHP  automated  IS:  The  accuracy  of  quality 
indicators  produced  through  an  MCO's/PHP's  automated  IS  is  a  reflection  of  three  phenomena: 

1)  the  soundness  of  the  algorithm  the  MCO/PHP  used  to  produce  quality  indicators  from  its  IS 

2)  the  integrity  (completeness  and  accuracy)  of  the  MCO's  /PHP's  IS  at  capturing  enrollee 
information;  and 

3)  the  accuracy  of  the  information  translated  from  source  documents  (e.g.,  an  enrollee's  medical 
record)  into  automated  data  in  the  MCO's/PHP's  IS. 
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The  soundness  of  the  algorithm  the  MCO/PHP  used  to  produce  quality  indicators  from  its  IS  is  to 
be  assessed  in  Step  6.  In  order  to  assess  the  integrity  of  the  MCO's  IS,  and  the  accuracy  of  the 
information  translated  from  source  documents  (e.g.,  an  enrollee's  medical  record)  into  automated 
data  in  the  MCO's/PHP's  IS,  the  EQRO  should  review  a  copy  of  an  assessment  of  the  MCO's 
/PHP's  IS  and  any  validations  of  MCO/PHP  encounter  data  that  the  State  has  produced.  These 
activities  are  described  in  APPENDIX  A  and  in  the  protocol,  Validating  MCO  Encounter  Data, 
and  would  typically  be  conducted  as  a  part  of  another  activity  conducted  by  the  EQRO  or  other 
organization  as  part  of  another  activity;  e.g.,  validating  encounter  data,  validating  performance 
measures,  or  assessing  an  MCO's  or  PHP's  compliance  with  standards  for  MCO/PHP  IS  specified 
by  the  State  Medicaid  agency  or  other  organization  such  as  a  private  accrediting  organization.  In 
order  to  use  this  information  to  help  verify  the  accuracy  of  reported  quality  indicators,  the  EQRO 
should  obtain  a  copy  of  a  recently  completed  assessment  of  the  MCO's/PHP's  IS  and  validation  of 
its  encounter  data  from  the  MCO/PHP,  the  State  Medicaid  agency,  or  other  organization  identified 
by  the  MCO/PHP.  In  the  event  that  no  current,  independent  assessment  of  an  MCO's/PHP's  IS  or 
encounter  data  exists,  the  State  may  want  to  contract  out  this  function  as  a  part  of  validating 
MCO/PHP  PIPs. 

Assessing  the  MCO's/PHP's  algorithm  together  with  the  integrity  of  the  MCO's/PHP's  IS  and 
encounter  data  should  provide  a  strong  indication  of  the  accuracy  of  the  MCO's/PHP's  reported 
quality  indicators. 


END  OF  PROTOCOL 
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PERFORMANCE  IMPROVEMENT  PROJECTS 
PROJECT  MONITORING  WORKSHEET 

i  

!  Use  this  worksheet  or  a  similar  worksheet  as  a  guide  when  validating  MCO/PHP  Performance 

\      Improvement  Projects.  Answer  all  questions  for  each  activity.  Refer  to  the  protocol  for 

detailed  information  on  each  area. 

ID  of  evaluator:  Date  of  evaluation:     /  / 


Demographic  Information 


MCO/PHP  Name  or  ID: 


Project  Leader  Name: 


Telephone  Number: 


Name  of  Performance  Improvement  Project: 


Dates  in  Study  Period: 


/ 


/ 


to 


/ 


/ 


Type  of  Delivery  System  (check  all  that  are  applicable) 


Staff  Model 
Network 
Direct  IPA 
IPA  Organization 


MCO 
PHP 


Number  of  Medicaid  Enrollees  in  MCO  or  PHP 
Number  of  Medicare  Enrollees  in  MCO  or  PHP 
Number  of  Medicaid  Enrollees  in  Study 
Total  Number  of  MCO  or  PHP  Enrollees  in  Study 


Number  of  MCO/PHP  primary  care  physicians 
Number  of  MCO/PHP  specialty  physicians  


Number  of  physicians  in  study  (if  applicable) 


I. 


ACTIVITY  1 :    ASSESS  THE  STUDY  METHODOLOGY 


Step  1. 


REVIEW  THE  SELECTED  STUDY  TOPIC(S) 


Component/Standard 


N 


N/A 


Comments 


1 . 1      Was  the  topic  selected  through  data 
collection  and  analysis  of 
comprehensive  aspects  of  enrollee 
needs,  care  and  services? 


HCFA/CMSO/QPMG/DQSM  -  September  11,  2000  DRAFT 


Page  27 


Attachment  1 


1 .2.     Did  the  MCO's/PHP's  PIPs,  over  time, 
address  a  broad  spectrum  of  key 
aspects  of  enrollee  care  and  services? 

1.3.     Did  the  MCO/PHP,  over  time,  select 

topics  in  clinical  and  non-clinical  focus 
areas? 

Step  2 :          REVIEW  THE  STUDY  QUESTION(S) 

2.1.     Was/were  the  study  question(s)  stated 
clearly  in  writing? 

Step  3 :          REVIEW  THE  SELECTED  STUDY  INDIC ATOR(S) 

3.1.     Did  the  study  use  objective,  clearly 
defined,  measurable  indicators? 

3.2.     Did  the  indicators  measure  changes  in 
health  status,  functional  status,  or 
enrollee  satisfaction,  or  valid  proxies  of 
these  outcomes? 

Step  4:          REVIEW  THE  IDENTIFIED  STUDY  POPULATION  ^ 

4. 1 .     Did  the  MCO/PHP  clearly  define  all 
Medicaid  enrollees  to  whom  the  study 
question  and  indicators  are  relevant? 

4.2      If  the  MCO/PHP  studied  the  entire 
population,  did  its  data  collection 
approach  capture  all  enrollees  to 
whom  the  study  question  applied? 

Step  5:          REVIEW  SAMPLING  METHODS 

5.1.     Did  the  sampling  technique  consider 
and  specify  the  true  (or  estimated) 
frequency  of  occurrence  of  the  event, 
the  confidence  interval  to  be  used,  and 
the  margin  of  error  that  will  be 
acceptable? 

5.2      Were  the  selected  confidence  interval 
and  margin  of  error  reasonable? 
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5.3      Did  the  MCO/PHP  employ  valid 
sampling  techniques? 

5.4.     Did  the  sample  contain  a  sufficient 
number  of  enrollees? 

Step  6:          REVIEW  DATA  COLLECTION  PROCEDURES 

6. 1 .     Did  the  study  design  clearly  specify  the 
data  to  be  collected  I 

6.2.     Did  the  study  design  clearly  specify  the 
sources  of  data? 

6.3.     Did  the  study  design  specify  a 

systematic  method  of  collecting  valid 
and  reliable  data  that  represents  the 
entire  population  to  which  the  study's 
indicators  apply? 

6.4.     Did  the  instruments  for  data  collection 
provide  for  consistent,  accurate  data 
collection  over  the  time  periods 
studied? 

6.5.     Did  the  study  design  prospectively 
specify  a  data  analysis  plan? 

6.6.     Were  qualified  staff  and  personnel 
used  to  collect  the  data? 

Step  7:          ASSESS  IMPROVEMENT  STRATEGIES 

7.1.     Were  interventions  related  to 

causes/barriers  identified  through  data 
analysis  and  QI  processes  undertaken  ? 

Step  8:          REVIEW  DATA  ANALYSIS  AND  INTERPRETATION  OF  STUDY  RESULTS 

8. 1 .     Was  an  analysis  of  the  findings 
performed  according  to  the  data 
analysis  plan? 

8.2.      Did  the  MCO/PHP  present  numerical 
PIP  results  and  findings  accurately  and 
clearly? 
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8.3.     Did  the  analysis  identify:  initial  and 
repeat  measurements,  statistical 
significance,  factors  that  influence 
comparability  of  initial  and  repeat 
measurements,  and  factors  that  threaten 
internal  and  external  validity? 

8.4      Did  the  analysis  of  study  data  include 
an  interpretation  of  the  extent  to  which 
its  PIP  was  successful  and  follow-up 
activities? 

Step  9:          ASSESS  WHETHER  IMPROVEMENT  IS  SIGNIFICANT 

9. 1 .     Was  the  same  methodology  as  the 
baseline  measurement,  used,  when 
measurement  was  repeated? 

9.2.  &  9.3.  Was  significant  improvement 
demonstrated? 

Step  10:        ACHIEVING  SUSTAINED  IMPROVEMENT 

10.1.    Was  sustained  improvement 

demonstrated  through  repeated 
measurements  over  comparable  time 
periods? 

ACTIVITY  2.          VALIDATING  STUDY  FINDINGS 

1 .        Were  the  findings  of  the  initial  study 
validated?  What  were  its  findings? 
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Record  any  additional  comments  pertinent  to  the  design  and/or  performance  of  the  study: 
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Performance  Improvement 

Pro  j  ects 


Performance  Improvement 

Projects 

Ruth  Martin 
Health  Care  Consulting 


Ruth  M.  Martin 

Health  Care  Consulting 


Ruth  Martin  is  an  independent  consultant  who  has  more  than  25  years  of 
experience  in  managed  care,  Medicaid  programs,  and  community  health 
programs.  Ms.  Martin  previously  was  Deputy  Director  of  the  Robert  Wood 
Johnson  Foundation's  Medicaid  Managed  Care  Program,  where  she  now  serves  as 
Senior  Program  Consultant.  Prior  to  working  with  the  Medicaid  Managed  Care 
Program,  she  was  executive  director  of  HEALTH  PLUS,  a  Medicaid  managed 
care  plan  sponsored  by  Lutheran  Hospital  in  Brooklyn,  New  York. 

Prior  to  that,  Ms.  Martin  was  President  of  Vermont  Managed  Care,  a 
physician/hospital  organization  sponsored  by  Fletcher-Allen  Health  System  (the 
University  of  Vermont  Academic  Medical  Center).  Ms.  Martin  has  provided 
numerous  consultations  to  the  federal  government,  community  health 
organizations,  and  managed  care  organizations.  She  is  an  administrative  surveyor 
for  the  National  Committee  for  Quality  Assurance  and  has  provided  consultation 
to  NCQA,  including  the  chapter,  "Focused  Quality-of-Care  Studies:  Selection, 
Design  and  Implementation,"  in  Health  Care  Quality  Improvement  Studies  in 
Managed  Care  Settings,  Design  and  Assessment:  A  Guide  for  State  Medicaid 
Agencies..  Ms.  Martin  holds  both  MBA  and  MPH  degrees  from  Columbia 
University. 


71  Wiggins  St. 

mail  pager:  888-813-0705 

Princeton.  NJ  08540 


Phone  &fax:  609-683-4339  Voice- 
E-mail:  mthmart, 
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Beth  C.  Tortolani 
Massachusetts  Division  of  Medical  Assistance 


I 


Beth  C.  Tortolani,  RN,  BS,  MS 


Professional  Sketch 
September  2000 


Ms.  Tortolani  works  for  the  Commonwealth  of  Massachusetts,  Division  of  Medical  Assistance. 
Ms.  Tortolani  has  been  the  Director  of  the  Primary  Care  Clinician  (PCC)  Plan  since  1994.  Prior 
to  becoming  the  Director  of  the  PCC  Plan  Ms.  Tortolani  worked  in  a  variety  of  positions  at  the 
Division  which  included  the  oversight  of  managed  care  development  projects,  quality 
management  initiatives  and  program  management. 

Prior  to  working  for  the  Commonwealth  Ms.  Tortolani  held  several  management  positions  in  the 
health  care  industry.  Ms.  Tortolani  is  a  Registered  Nurse  and  holds  a  Bachelors  Degree  in 
Nursing  from  the  University  of  Connecticut,  Storrs  Connecticut,  and  a  Masters  Degree  in 
Management  from  Lesley  College,  Cambridge,  Massachusetts. 


Address 

Division  of  Medical  Assistance 
600  Washington  Street  -  5th  Floor 
Boston,  Massachusetts  02 1 1 1 

Email    Btortol  an  i  (a)  nt  .dma .  state .  ma.  us 

Phone  617-210-5402 
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Diane  Herr 

Maryland  Health  Choice  and  Acute  Care  Administration 


Diane  Herr 


Diane  Herr  is  currently  the  Deputy  Director  for  Provider  Management  of  the  HealthChoice  Program 
for  the  Maryland  Medicaid  Program.  Ms.  Herr  has  been  a  Deputy  Director  in  the  Medicaid  Program 
for  the  last  6  years.  She  has  held  various  management  positions  during  her  16  year  tenure  at  the 
Maryland  Department  of  Health  and  Mental  Hygiene  including  3  years  as  Executive  Assistant  to  the 
Maryland  Secretary  of  Health.  Ms  Herr  has  played  an  integral  part  in  the  planning,  development, 
implementation  and  management  of  Maryland's  HealthChoice  Managed  Care  Program  and  her 
current  position  includes  responsibility  for  contract  management  and  all  quality  assurance  activities 
of  HealthChoice  as  well  as  management  of  the  Hospital,  Physician,  and  Pharmacy  programs  within 
the  Medicaid  fee  for  service  arena. 
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MC  W516 

Austin,  TX  78751 

Phone:  (512)438-5139 

Fax:  (512)438-2845 

Email:  jerry.driskell@dhs.state.tx.us 
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Denise  M.  Dugan 

Quality  Management  Administrator 
Division  of  Social  Services  -  Medicaid 
1901  North  Dupont  Highway 
P.O.  Box  906,  Lewis  Building 
Newcastle,  DE  19720 
Phone:     (302)  577-4880 
Fax:        (302)  577-4899 
Email:     ddugan@state.de. us 

Deirdre  Duzor 

Director,  Division  of  Quality  System  Management 

Health  Care  Financing  Administration 

Division  of  Quality  System  Management 

7500  Security  Boulevard 

MS:  S3-13-15 

Baltimore,  MD  21244-1850 

Phone:     (410)  786-4626 

Fax:  (410)786-1008 

Email:  dduzor@hcfa.gov 

Kathe  R.  Eastham 

Registered  Nurse,  BSN 
Oklahoma  Health  Care  Authority 
4545  North  Lincoln  Boulevard 
S124 

Oklahoma  City,  OK  73105 
Phone:  (405)522-7155 
Fax:  (405)  522-7378 
Email:  easthamk@ohca.state.ok.us 

David  K.  Ellison 

Program  Consultant 

Health  Resources  and  Services  Administration 
Midwest  Field  Office  -  Kansas  City 
601  East  12th  Street 
Room  1728 

Kansas  City,  MO  64106 
Phone:  (816)426-5296 
Fax:  (816)426-3633 
Email:  dellison@hrsa.gov 


Chris  M.  English 

Quality  Manager 
SRS  Health  Care  Policy 
Managed  Care 
915  SW  Harrison 
Room  651  South 
Topeka,  KS  66612 
Phone:     (785)  2964750 
Fax:  (785)296-4813 
Email:  cme@srs.kansas.org 

Anna  Fallieras 

Deputy  Chief  of  Staff 

Department  of  Health  and  Mental  Hygiene 

Health  Care  Financing 

201  West  Preston  Street 

5th  Floor 

Baltimore,  MD  21201 
Phone:     (410)  767-5954 
Fax:  (410)333-7687 

Kathleen  M.  Farrell 

Project  Officer 

Health  Care  Financing  Administration 

7500  Security  Boulevard 

MS:  S2-03-18 

Baltimore,  MD  21207 

Phone:  (410)786-1236 

Fax:        (410)  786-8534 

Email:  kfarrell@hcfa.gov 

Matthew  O.  Ferrara 

Administrator 

Texas  Community  on  Alcohol  and  Drug  Abuse 

NorthSTAR  Provider  and  Enrollee  Services 

909  West  45th  Street 

Austin,  TX  78711-2668 

Phone:     (512)  206-5470 

Fax:  (512)206-5383 

Email:      matthew.ferrara@mhmr.  state,  tx.us 
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Terri  Fey 

Health  Insurance  Specialist 

Health  Care  Financing  Administration 

233  North  Michigan  Avenue 

Chicago,  IL  60601 

Phone:  (312)353-8168 

Fax:  (312)886-2303 

Email:  tfey@hcfa.gov 


Karen  Ford 

Health  Program  Manager 
Utah  Department  of  Health 
Division  of  Health  Care  Financing 
288  North  1460  West 
P.O.  Box  143108 
Salt  Lake  City,  UT  84114-3108 
Phone:     (801)  538-6637 
Fax:  (801)538-6099 
Email:      kford@doh. state. ut.us 

Alan  F.  Freund 

Managed  Care  Coordinator 

Health  Care  Financing  Administration 

Region  V 

233  North  Michigan  Avenue 
Suite  600 
Chicago,  IL  60601 
Phone:     (312)  353-1589 
Fax:  (312)353-3866 
Email:  afreund@hcfa.gov 

Ann  M.  Froio 

Chief,  Bureau  of  Quality  Management 
Arizona  Department  of  Health  Services 
Division  of  Behavioral  Health  Services 
2122  East  Highland 
Suite  100 

Phoenix,  AZ  85016 
Phone:     (602)  553-9062 
Fax:        (602)  553-9144 
Email:      afroio@hs. state. az.us 


Michaela  L.  Funaro 

Program  Manager 

Iowa  Department  of  Human  Services 

Division  of  Medical  Services 

Hoover  Building 

5th  Floor 

Des  Moines,  IA  50319 

Phone:  (515)281-5189 

Fax:        (515)  281-6230 

Email:  mfunaro@dhs.state.ia.us 

Judith  Gallagher 

Director 

Maternal,  Child,  Community  Health 
Health  Systems  Research,  Inc 
1200  18th  Street,  NW 
Room  700 

Washington,  DC  20007 
Phone:  (202)828-5100 
Fax:        (202)  728-9469 
Email:  jgallagher@hsrnet.com 

Jorge  Garcia-Ortiz 

Executive  Assistant 

Puerto  Rico  Health  Insurance  Administration 
Calle  de  Tapjuan 
Suite  253 

Old  San  Juan,  PR  00902-4264 
Phone:     (787)  725-9427 
Fax:        (787)  724-4966 
Email:  jgarcia@ases.prstar.net 

Sharon  M.  Gilles 

Health  Care  Policy  Analyst 

Health  Care  Financing  Administration 

Center  for  Medicaid  State  Operations/QPMG/DQSM 

7500  Security  Boulevard 

MS:  S3-13-15 

Baltimore,  MD  21244-1850 
Phone:  (410)786-1177 
Fax:        (410)  786-1008 
Email:  sgilles@hcfa.gov 
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J.  Samuel  Griswold 

Director 

South  Carolina  Department  of  Health  and  Human  Services 
1801  Main  Street 
11th  Floor 

Columbia,  SC  29201 
Phone:     (803) 898-2504 
Fax:  (803)8984515 


Paula  F.  Gullion 

Quality  Assurance  Research  Analyst 
Oklahoma  Health  Care  Authority 
4545  North  Lincoln  Boulevard 
Suite  124 

Oklahoma  City,  OK  73150 
Phone:  (405)522-7170 
Fax:  (405)  522-7378 
Email:  paulagullion@ohca.state.ok.us 

Wanda  M.  Gutierrez 

Quality  Improvement  Specialist 

Utah  Department  of  Health 

Division  of  Health  Care  Financing 

288  North  1460  West 

P.O.  Box  143108 

Salt  Lake  City,  UT  84114-3108 

Phone:  (801)538-9484 

Fax:        (801)  536-0154 

Email:     wgutierr@doh. state. ut.us 

Mary  R.  Guy 

Health  Insurance  Specialist 

Health  Care  Financing  Administration 

Center  for  Medicaid  and  State  Operations 

7500  Security  Boulevard 

MS:  S2-14-25 

Baltimore,  MD  21244 

Phone:  (410)786-2772 

Fax:  (410)786-9004 

Email:  mguy@hcfa.gov 


Marcy  R.  Hajdukiewicz 

Program  Administrator 

2727  Mahan  Drive 

Fort  Knox,  B-1,  MS:  26 

Tallahassee,  FL  32308 

Phone:     (850)  487-0640 

Fax:        (850)  488-0073 

Email:  hajudkim@fdhc.state.fl.us 


Thomas  Hamilton 

Director 

Health  Care  Financing  Administration 

Disabled  and  Elderly  Health  Program  Group 

7500  Security  Boulevard 

Baltimore,  MD  21244-1850 

Phone:  (410)786-6763 

Fax:  (410)786-9004 

Email:  thamilton@hcfa.gov 

Gary  E.  Harbison 

Department  of  Mental  Health  Outcomes  Coordinator 

Missouri  Department  of  Health 

P.O.  Box  887 

Jefferson  City,  MO  65102 

Phone:     (573)  751-8917 

Fax:        (573)  751-8917 

Email:     mz_harbg@mail.dmh. state. mo.us 


Kathleen  Hastings 

Regional  Public  Health  Consultant 

Health  Resources  and  Services  Administration 

Denver  Field  Office 

1961  Stout  Street 

Room  409 

Denver,  CO  80294 

Phone:     (303)  844-7861 

Fax:        (303)  844-0002 

Email:  khastings@hrsa.gov 
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Chris  Heldman 

Senior  Program  Analyst 
65  East  State  Street 
3rd  Floor 

Columbus,  OH  43266 

Phone:     (614)  4664693 

Fax:        (614)  728-4516 

Email:  heldmr@odjfs.state.oh.us 


Lowware  Holliman 

Operations  Manager 

1300  Strawberry  Square 

Harrisburg,  PA  17120 

Phone:  (717)783-1437 

Fax:  (717)705-1643 

Email:  lhollima@ins.state.pa.us 


Mary  Herkert 

OMC II 

Agency  for  Health  Care  Administration 

6800  North  Dale  Mabry  Highway 

Suite  220 

Tampa,  FL  33614 

Phone:  (813)871-7600 

Fax:  (813)871-8313 

Email:  herkertm@fdhc.state.fl.us 


Vicky  Hosey 

Public  Service  Administrator 

Illinois  Department  Public  Aid 

1011  South  Second  Street 

Springfield,  IL  62704 

Phone:  (217)557-1718 

Fax:        (217)  557-8604 

Email:  aidd0307@mail.idpa.state. 


Melba  R.  Hinojosa 

Health  Program  Specialist 
714  P.  Street 
Room  650 

Sacramento,  CA  95814 
Phone:  (916)654-0748 
Fax:  (916)654-0288 
Email:  mhinojos@dhs.ca.gov 


Katrina  Howard 

Policy  Procedure  Analyst  1 
FSSA  /OMPP 

402  West  Washington  Street 

Room  W382  MS07 

Indianapolis,  IN  46204 

Phone:     (317)  232-2034 

Fax:  (317)232-7382 

Email:      khoward@fssa.state.in. us 


JoAnne  Hoesel 

Administrator 

Division  of  Mental  Health  and  Substance  Abuse 
Children's  Mental  Health  Programs 
Room  302,  State  Capital  -  Judicial  Wing 
600  E.  Boulevard  Avenue 
Bismark,  ND  58505 
Phone:  (701)328-2335 
Fax:        (701)  328-3538 
Email:  sohoej@state.nd.us 


Ruth  Hughes 

Medicaid  Program  Representative 
Health  Care  Financing  Administration 
Region  V 

233  North  Michigan 
Suite  600 
Chicago,  IL  60601 
Phone:  (312)353-1670 
Fax:  (312)353-3866 
Email:  rhughes@hcfa.gov 
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Todd  Hurt 

Program  Manager 
Region  III  DHW  Medicaid 
Division  of  Mental  Health 
3402  Franklin  Road 
Caldwell,  ID  83605 
Phone:     (208)  459-0092 
Fax:  (208)454-7714 
Email:  hurtt@idhw.state.id.us 

Brenda  E.  Jackson 

Health  Insurance  Specialist 

Health  Care  Financing  Administration 

601  East  12th  Street 

Kansas  City,  MO  64106 

Phone:  (816)426-3406 

Fax:  (816)426-3851 

Email:  bjackson1@hcfa.gov 


Gaylene  R.  Jeffries 

Quality  Improvement  Manager 

Nebraska  State  Medicaid 

301  Centennial  Mall  South 

Lincoln,  NE  68509 

Phone:  (402)471-9415 

Fax:  (402)471-9092 

Email:     gaylene.jeffries@hhss. state. ne. us 


Helene  S.  Jo 

Managed  Care  Nurse  Supervisor 
Med-QUEST  Division 
601  Kamokila  Boulevard 
Suite  506B 
Kapolei,  HI  96707 
Phone:  (808)692-8111 
Fax:  (808)692-8131 


Steven  M.  Johnson 

Program  Management  Officer 

New  Jersey  Department  of  Health  and  Senior 

Division  of  Addiction  Services 

P.O.  Box  362 

Trenton,  NJ  08625-0362 

Phone:     (609)  984-9896 

Fax:  (609)292-3816 

Email:      smjohnson@doh.  state,  nj.  us 

Hilary  Jones 

Medicaid  Services  Specialist  III 
Department  of  Human  Resources 
DHCP 

1100  E.  William  Street 
Suite  204 

Carson  City,  NV  89701 

Phone:     (775)  687-4176 

Fax:        (775)  684-8792 

Email:      hjones@govmail. state. nv.us 

Marineda  Jung 

Managed  Care  Operations  Manager 

Missouri  Department  of  Social  Services 

615  Howerton  Court 

Jefferson  City,  MO  65109 

Phone:  (573)751-5178 

Fax:        (573)  526-3946 

Email:  jackiejung@mail.medicaid.state.us 


Diana  Kading 

Managed  Health  Care  Manager 
Iowa  Department  of  Human  Services 
Division  of  Medical  Services 
Hoover  State  Building 
5th  Floor 

Des  Moines,  IA  50319 
Phone:     (515)  281-5394 
Fax:       (515)  281-6230 
Email:  dkading@dhs.state.ia.us 
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Joan  M.  Kapowich 

Program  and  Policy  Manager 
Office  of  Medical  Assistance  Program 
500  Summer  Street,  NE 
E-35 

Salem,  OR  97301-1077 

Phone:     (503)  945-6500 

Fax:        (503)  373-7689 

Email:  joan.m.kapawich@state.or.us 

Laurel  Karabatsos 

Manager 
HCPF 

Quality  Assurance  Section 
1575  Sherman  Street 
5th  Floor 

Denver,  CO  80203 

Phone:     (303)  866-6320 

Fax:        (303)  866-2573 

Email:     laurel,  karabatsos@state.co.  us 

Neva  J.  Kaye 

Director 

Medicaid  Managed  Care  Resource  Center 
National  Academy  for  State  Health  Policy 
50  Monument  Square 
Suite  502 

Portland,  ME  04101 
Phone:     (207)  874-6524 
Fax:        (207)  874-6527 
Email:  nkaye@nashp.org 

Sharon  M.  Kernan 

Assistant  Administrator/Registered  Nurse,  MPH 

Rhode  Island  Department  of  Human  Services 

Community  and  Planning  Services 

600  New  London  Avenue 

Cranston,  Rl  02920 

Phone:  (401)462-3392 

Fax:  (401)462-6353 

Email:      sharonk@gw.dhs. state. ri. us 


Diona  M.  Kristian 

Project  Officer 

Health  Care  Financing  Administration 

7500  Security  Boulevard 

MS:  S2-02-06 

Baltimore,  MD  21244 

Phone:     (410)  786-3283 

Fax:        (410)  786-8534 

Email:  dkristian@hcfa.gov 

Ira  Lacy 

Director,  Managed  Care  Services 
Tennessee  Department  of  Health 
Bureau  of  Alcohol  and  Drug  Abuse  Services 
Cordell  Hull  Building  425,  5th  Avenue  North 
3rd  Floor 

Nashville,  TN  37247-4401 
Phone:  (615)741-1921 
Fax:  (615)532-2419 
Email:  ilacy@mail.state.tn.us 

Deyle  Randall  Lee 

Senior  Data  Analyst 

Colorado  Alcohol  and  Drug  Abuse  Division 
4055  South  Lowell  Boulevard 
Building  KA 

Denver,  CO  80236-3120 


Hyesun  Lee 

Health  Care  Financing  Administration 
233  North  Michigan  Avenue 
Suite  600 
Chicago,  IL  60601 
Phone:  (312)353-1565 
Fax:  (312)353-3866 
Email:  hlee2@hcfa.gov 
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Robert  E.  Lee 

Executive  Director 

Central  Oklahoma  Community  Health  Center 

909  E.  Alameda 

P.O.  Box  400 

Norman,  OK  73071 

Phone:     (405)  573-3901 

Fax:        (405)  573-3958 

Email:  rlee@odmhsas.org 

Tracy  E.  Leeper 

Grant  Projects  Manager 
1200  NE  13th  Street 
Oklahoma  City,  OK  73117 
Phone:     (405)  522-3822 
Fax:        (405)  522-3650 
Email:  tleeper@odmhsas.org 


Jacqueline  Lloyd 

Posf  Doctoral  Fellow 

Center  for  Substance  Abuse  Treatment  Office  of  Manged  Care 
Substance  Abuse  and  Mental  Health  Services  Administration 
5515  Security  Lane 
RW  II,  Suite  740 
Rockville,  MD  20857 
Phone:  (301)443-2341 
Fax:  (301)443-3543 
Email:  jlloyd@samhsa.gov 

James  MacLeod 

Quality  Assurance  Unit  Manager 

Oregon  Medicaid  Health  Services 

2575  Bittern  Street,  NE 

P.O.  Box  14250 

Salem,  OR  97309-0740 

Phone:     (503)  945-9714 

Fax:        (503)  373-7327 

Email:  mac!eodj@mail.mhd.hr.state.or.us 


Paula  R.  Margolis 

Managed  Care  Quality  Analyst 
State  Medicaid 
600  East  Broad  Street 
Suite  1140 

Richmond,  VA.  23219 

Phone:  (804)371-6060 

Email:      pmargoli@dmas. state. va. us 


Leslie  A.  Martin 

Department  Head 

Department  of  Health  and  Human  Services 

Department  of  Medical  Support 

P.O.  Box  8206 

Columbia,  SC  29202-8206 

Phone:     (803)  898-2805 

Fax:  (803)898-4511 

Email:      martinl@dhhs.  state,  sc.usa 

Mona  L.  Martin 

Social  Work  Consultant 

Health  Resources  and  Services  Administration 

26  Federal  Plaza 

Room  3337 

New  York,  NY  10278 

Phone:     (212)  264-4625 

Fax:  (212)264-2673 

Email:  mcmartinv@hrsa.gov 

Pam  Mason 

Medicaid  Managed  Care  Program  Development  Manager 
DHW  Medicaid 

3380  Americana  Terrace,  Suite  230 

P.O.  Box  83720 

Boise,  ID  83720-0036 

Phone:  (208)364-1890 

Fax:  (208)364-1846 

Email:  masonpa@mmis.state.id.us 
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Paul  I.  Masotti 

Program  Planning  Analyst  V 

1  West  Wilson  Street 

Madison,  Wl  53702 

Phone:     (608)  264-7724 

Fax:  (608)261-7792 

Email:  masotpi@dhfs.state.wi.us 


Sheryl  A.  Mathis 

Project  Manager 
HSR,  Inc. 

1200  18th  Street,  NW 
Suite  700 

Washington,  DC  20036 
Phone:  (202)828-5100 
Fax:        (202)  728-9469 
Email:  smathis@hsrnet.com 

Darendia  L.  McCauley 

Director  of  Quality  Assurance 
Oklahoma  Health  Care  Authority 
4545  North  Lincoln  Boulevard 
Suite  #124 

Oklahoma  City,  OK  73013 
Phone:  (405)  522-7355 
Fax:  (405)  522-7378 
Email:  mccauled@ohca.state.ok.us 

Brenda  McCormick 

Maine  PnmeCare  Program  Manager 

Bureau  of  Medical  Services/Department  of  Health  and 

Human  Services 

State  House  Station  #1 1 

Augusta,  ME  04333-001 1 

Phone:  (207)287-1774 

Fax:        (207)  287-2675 

Email:      brenda. mccormick@state. me. us 


Roxane  McCormick 

Nurse  Consultant 
Texas  Department  of  Health 
Bureau  of  Managed  Care 
1100  West  49th  Street 
MC  Y-980 

Austin,  TX  78756-3168 

Phone:     (512)  338-6438 

Fax:        (512)  338-6969 

Email:      roxane. mccormick@tdh. state. tx. us 

Eileen  McGorry 

Quality  Manager 

North  Star 

909  West  45th 

P.O.  Box  12668 

Austin,  TX  78711 

Phone:     (512)  206-5460 

Fax:        (512)  206-5383 

Email:  eileen.mcgorry@mhmr.state.tx.us 

Michael  Melendez 

Health  Insurance  Specialist 

Health  Care  Financing  Administration 

26  Federal  Plaza 

Room  3800 

New  York,  NY  10278 

Phone:  (212)264-9121 

Fax:        (212)  264-6814 

Email:  mmelendez@hcfa.gov 

Roland  M.  Mena 

Bureau  Chief 
3467  Hannibal 
Butte,  MT  59701 
Phone:     (406)  444-3907 
Fax:        (406)  4444435 
Email:  rmena@state.mt.us 


14 


 -j 


Brian  Miller 

Division  of  Mental  Health 
120  North  200  West 
'Salt  Lake  City,  UT  84103 
Phone:  (801)538-4270 
Fax:  (801)538-9892 


MaryAnn  Miller 

Director 

DMHAS/QAI 

Quality  Management 

410  Capitol  Avenue 

Hartford,  CT  06134 

Phone:  (860)418-6738 

Fax:  (860)418-6896 

Email:      maryann.miller@po.  state,  ct.  us 

Robert  O.  Miller 

Operations  Manager 

Office  of  Alcohol  and  Drug  Abuse  Programs 

500  Summer  Street,  NE 

E-86 

Salem,  OR  97301-1118 
Phone:     (503)  945-6185 
Fax:        (503)  378-8467 
Email:  bob.miller@state.or.us 

Linda  S.  Murphy 

Medicaid  Policy  Analyst 

Health  Care  Financing  Administration 

7500  Security  Boulevard 

MS:  S3-13-15 

Baltimore,  MD  21244 

Phone:  (410)786-0435 

Fax:  (410)786-1008 

Email:  lmurphy@hcfa.gov 


Sen-Yoni  Musingo 

Database  Administrator 

Florida  Department  of  Children  and  Families 

Child  Mental  Health 

1317  Winewood  Boulevard 

Tallahassee,  FL  32301-0900 

Phone:  (850)413-7171 

Fax:  (850)413-6886 

Email:      senyoni_musingo@dcf.state.fi. us 

Randy  W.  Myers 

Director 

West  Virginia  Department  of  Health 
Bureau  for  Medical  Services 
350  Capitol  Street 
Room  251 

Charleston,  WV  25301-3708 
Phone:     (304)  558-5974 
Fax:        (304)  558-4398 
Email:  randymyers@wvdhhr.org 

Mary  O.  Norris 

Policy  Supervisor 

1201  Capitol  Access  Road 

Baton  Rouge,  LA  70821-9030 

Phone:  (225)342-1247 

Fax:        (225)  342-1411 

Email:      mnorhs@dhhmail.dhh. state. la.us 


D.  Stephen  Nose 

Health  Insurance  Specialist 
Health  Care  Financing  Administration 
1600  Broadway 
Suite  700 

Denver,  CO  80202 
Phone:     (303) 844-7045 
Fax:        (303)  844-3753 
Email:  snose@hcfa.gov 
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Lori  N.  Nuebel 

Administrator 
SRS  Health  Care  Policy 
Quality  Enhancement 
915  S.W.  Harrison 
5th  Floor  North 
Topeka,  KS  66612 
Phone:     (785)  296-3471 
Fax:        (785)  296-6142 
Email:  lnn@srs.kansas.org 

Ann  E.K.  Page 

Technical  Director 

Health  Care  Financing  Administration 

Division  of  Quality  System  Management 

7500  Security  Boulevard 

MS:  S3-13-15 

Baltimore,  MD  21244-1850 

Phone:  (410)786-0435 

Fax:        (410)  786-1008 

Sharon  S.  Patterson 

Nurse  Consultant 
Division  of  Medicaid 
427  West  62nd  Street 
Kansas  City,  MO  64113 
Phone:  (816)426-3406 
Fax:  (816)426-3851 
Email:  spatterson@hcfa.gov 


Rebecca  R.  Paul 

State  Child  Health  Insurance  Program  Coordinator 

Health  Care  Financing  Administration 

75  Hawthorne  Street 

San  Francisco,  CA  94105 

Phone:     (415)  744-3553 

Fax:  (415)744-2933 

Email:  rpaul@hcfa.gov 


Amy  A.  Pollreisz 

Program  Specialist 

Department  of  Human  Services 

Division  of  Mental  Health,  Child  and  Adolescent  Services 

3800  E.  Highway  34 

Hillsview  Plaza 

Pierre,  SD  57501 

Phone:     (605)  773-5991 

Fax:        (605)  773-7076 

Email:  amy.pollreisz@state.sd.us 

Scott  Ponaman 

William  M.  Mercer,  Inc. 
3131  East  Camelback  Road 
Suite  300 

Phoenix,  AZ  85016 

Phone:     (602)  522-6553 

Fax:        (602)  957-9573 

Email:  scott.ponaman@us.wmmercer.com 


Lorin  Ranbom 
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